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Executive Summary 

Minnesota Reading Corps is the largest AmeriCorps State program in the country. Its mission is to help every Minnesota child 
become a proficient reader by the end of third grade. To meet this goal, the Minnesota Reading Corps and its host organization, 
ServeMinnesota Action Network, engages a diverse group of AmeriCorps members to provide evidence-based literacy 
enrichment and tutoring services to preschool (PreK) students and at-risk Kindergarten through third grade (K-3) elementary 
school students. 

This report, commissioned by the Corporation for National and Community Service (CNCS), describes the results of a quasi- 
experimental design (QED) outcome evaluation of 1,534, 3-, 4-, and 5-year old PreK students who attended either a Minnesota 
Reading Corps PreK program site or a comparison site during the 2013-2014 school year. Students were enrolled at 25 
Minnesota Reading Corps PreK sites and 25 comparison PreK sites matched to each Minnesota Reading Corp on specific site- 
level characteristics. The goal of the outcome evaluation was to determine the effects of the Minnesota Reading Corps program 
on PreK students’ emergent literacy skills. 

Key Study Findings 

PreK students in the Minnesota Reading Corps program achieved significantly higher emergent literacy 
assessment scores by the end of the school year than did students enrolled at matched-comparison sites. 

The 4- and 5-year old students at Minnesota Reading Corps PreK sites outperformed students at matched-comparison sites on 
all five emergent literacy outcomes assessed: letter sound and letter name fluency (alphabet knowledge), rhyming and 
alliteration fluency (phonological awareness), and picture name fluency (vocabulary). The effect sizes associated with these 
differences were not only significant, but substantial in magnitude. Further, 4- and 5-year old Minnesota Reading Corps PreK 
students met or exceeded end of school year targets for all five emergent literacy outcomes, suggesting that they were 
Kindergarten-ready on these critical skills. 

Three-year old students at Minnesota Reading Corps PreK sites outperformed students at matched-comparison sites on two of 
four emergent literacy outcomes measured: rhyming fluency (phonological awareness) and picture name fluency (vocabulary). 
The effect sizes associated with these differences were not only significant, but substantial in magnitude. Three-year old 
students’ scores on the remaining two outcomes, letter name recognition (alphabet knowledge) and alliteration fluency 
(phonological awareness), grew significantly during the school year, but not significantly more than students in comparison 
classrooms. The results suggest that 3-year old students enrolled in Minnesota Reading Corps PreK classrooms make 
meaningful growth in key emergent literacy outcomes associated with later reading comprehension. 

The Minnesota Reading Corps PreK program was equally effective for PreK students regardless of gender, 
race/ethnicity or Dual Language Learner (DLL) status. 

Students from all subgroups examined produced significant growth in emergent literacy outcomes regardless of gender, 
race/ethnicity, or DLL status. In instances where statistically significant differences were found, students at Minnesota Reading 


OUTCOME EVALUATION OF THE MINNESOTA READING CORPS PREK PROGRAM 


Page v 


The Corporation for National and Community Service \ 2015 


Corps PreK sites traditionally at-risk for academic failure (i.e., DLLs, non-White students) grew more over the course of the year 
than similar types of students at comparison sites. 

The Minnesota Reading Corps PreK program was effective across multiple sites and site types; thus, the 
program is highly replicable in multiple preschool settings. 

The Minnesota Reading Corps PreK program was consistently effective across the individual sites where the students received 
program services, regardless of the type of school setting in which the program was implemented. The lack of school-level 
effects on student outcomes validates the Minnesota Reading Corps’ approach to training, coaching and supervision. 

About the Minnesota Reading Corps 

The Minnesota Reading Corps program was started in 2003 to provide emergent literacy enrichment and tutoring to students in 
four Head Start programs. In 2005, it expanded to serve elementary school students in Kindergarten through third grade (K-3). 
Today, the Minnesota Reading Corps is the largest AmeriCorps State program in the country. The core activities of the program, 
and its host organization, ServeMinnesota Action Network, are to recruit, train, place and monitor AmeriCorps members who 
implement evidence-based literacy interventions for at-risk K-3 students and preschool children. 

AmeriCorps members in the Minnesota Reading Corps program serve in school-based settings to implement Minnesota Reading 
Corps literacy enrichment strategies and conduct interventions with PreK-3 students using a Response to Intervention (Rtl) 
framework. The key features of the Minnesota Reading Corps Rtl framework are: 

■ Clear literacy targets at each age level from PreK through grade 3 

■ Benchmark assessment three times a year to identify students eligible for one-on-one or small group interventions 

■ Scientifically based interventions 

■ Frequent progress monitoring during intervention delivery 

■ High-quality training and coaching in program components, and literacy assessment and instruction 

In the Rtl framework, data play the key roles of screening students' eligibility for services and then monitoring students’ progress 
towards achieving academic goals (i.e., targets). The Minnesota Reading Corps screens students for program eligibility three 
times a year (i.e., Fall, Winter, Spring) with two sets of grade-specific, literacy-focused general outcome measures (i.e., IGD1 1.0 
for PreK and FAST for K-3) that possess criterion-referenced grade- and content-specific performance benchmarks. Program 
staff use scores from these general outcome measures to categorize students into one of three possible tiers (i.e., proficiency 
levels; see Figure 1): Tier 1 students score at or above benchmark and benefit from typical classroom instruction (75-80% of 
students score in this category); Tier 2 students score below benchmark and require specific supplemental interventions until 
they meet benchmarks (15-20% of students fall into this category); and Tier 3 students require intensive intervention provided by 
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a special education teacher or literacy specialist and often have individualized educational plans (5-10% of students qualify for 
this category). 1 

Figure 1 . Minnesota Reading Corps Response to Intervention Tiers 



TIER 1 

Whole class, high-quality instruction (Pre-K) 
75%-80% 


The Minnesota Reading Corps PreK program includes both an immersive “push-in” component, where members provide whole- 
class literacy enrichment for all students (i.e., Tier 1), as well as targeted small group and one-to-one component, where 
members provide more individualized interventions to students struggling with emergent literacy skills (i.e., Tiers 2 and 3). The 
PreK program focuses on integrating the “Big Five” Essential Early Literacy Predictors outlined by the National Early Literacy 
Panel (2008) 2 into all aspects of the daily classroom routine. The “Big Five” for preschool students include: 1) conversational 
skills, 2) vocabulary and background knowledge, 3) book and print rules, 4) phonological awareness (i.e., rhyming and 
alliteration), and 5) alphabet knowledge (i.e., letter name recognition and letter sound correspondence). Classroom teachers and 
AmeriCorps members are tasked with creating a Literacy Rich Classroom using evidence-based practices assessed in the 
ELLCO (Early Language & Literacy Classroom Observation) tool and implementing a Literacy Rich Schedule. In addition, 
AmeriCorps members serving in the PreK program are responsible for enacting the SEEDS of Emergent Literacy, 3 which 
provides both members and teachers with specific strategies to enhance literacy instruction for all children in the classroom. 


1 The percentage of children testing into Tiers 1-3 are national estimates based on the literature: 
http://www.rtinetwork.org/essential/tieredinstruction/tiered-instruction-and-intervention-rti- 

model ; http://www.hanoverresearch.com/insiqhts/developinq-a-maximum-impact-response-to-intervention-proqram/7Fk-1 2-education : Hoover, 
J.J., & Patton, J.R. (2008). The role of special educator in a multi-tiered instructional system. Intervention in School and Clinic, 43(4), 195-202. 

2 National Early Literacy Panel. (2008). Developing early literacy: Report of the National Early Literacy Panel. Washington, DC: National 
Institute for Literacy. 

3 Horst, K., & Passe, A. (2004). Creatine Literacy Rich Classrooms for Preschool Children (Apes 0-5) . Presented at the 2004 CEED 
Symposium. Minneapolis, MN. 
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Outcome Evaluation Methodology 


The PreK outcome evaluation is one of several complementary studies of the Minnesota Reading Corps program: a process 
assessment of program implementation in 20 PreK and K-3 sites; 4 a Phase II process assessment focused solely on the PreK 
program 5 ; a randomized controlled trial (RCT) impact evaluation of the program on K-3 students’ literacy outcomes; 6 and a 
survey of AmeriCorps members. The PreK outcome evaluation focused on the following three research questions: 

1 . What is the effect of the Minnesota Reading Corps program on student literacy outcomes? 

2. Does the effect on literacy outcomes vary by student characteristics/demographics? 

3. Do effects on literacy outcomes vary by site/school? 

The methodology used was informed by the 2008-2009 and 2010-201 1 annual Minnesota Reading Corps evaluations. 7 

School and Student Selection 

The site selection process was designed to ensure representation of the diverse schools participating in the Minnesota Reading 
Corps PreK program. Therefore, site selection was both purposive and involved a multi-step stratified sampling process. As it 
was not our intention to assess how well sites adopted/executed the program during their first year of participation, the site 
sampling process began by limiting the sampling frame to only sites that had fully implemented the Minnesota Reading Corps 
PreK program for at least one year. Next, to ensure geographical diversity, sites were stratified by urbanicity (i.e., urban, 
suburban, and rural), institution type (i.e., public preschool, community-based program and Head Start center), and student ages 
served within a classroom (i.e., 4- and 5-year old students only ora mix of 3-, 4-, and 5-year old students). A total of 40 sites 
were selected, using Probability Proportional to Size (PPS), whereby larger sites based on student enrollment (defined as the 
number of PreK students served by the Minnesota Reading Corps during the 2012-2013 school year) had a higher probability of 
selection. Using PPS ensured a statistically adequate sample size to conduct the PreK outcome evaluation. It is important to 
note that while the Minnesota Reading Corps program staff encouraged sites to participate, participation in the evaluation was 
voluntary. In the end, 39 sites agreed to participate in the PreK outcome evaluation during the 2013-2014 school year. 8 

To identify comparison sites for the PreK QED, the study followed U.S. Department of Education's Institute of Education 
Sciences’ (IES) What Works Clearing House (WWC) guidelines for a QED that “Meets Evidence Standards with Reservations.” 
PreK programs were eligible for consideration for the comparison group if: 1) the program had no history of implementing the 
Minnesota Reading Corps program; and 2) the program staff had not been trained in the SEEDS of Emergent Literacy 


4 Hafford, C., Markovitz, C., Hernandez, M, et al. (February 2013). Process Assessment of the Minnesota Reading Corps Program. (Prepared 
under contract to the Corporation for National and Community Service). Chicago, IL: NORC at the University of Chicago. 

5 Hafford, C., Markovitz, C., Hernandez, M, etal. (October 2014). Phase it Process Assessment of the Minnesota Reading Corps Program. 
(Prepared under contract to the Corporation for National and Community Service). Chicago, IL: NORC at the University of Chicago. 

6 C., Markovitz, C., Hernandez, M, et al. (March 2014). Impact Evaluation of the Minnesota Reading Corps K-3 Program. (Prepared under 
contract to the Corporation for National and Community Service). Chicago, IL: NORC at the University of Chicago. 

7 Bollman, K. & Silberglitt, B. (2009). Minnesota Reading Corps Final Evaluation 2008-2009. Minneapolis, MN: MRC.; Bollman, K. & Silberglitt, 
B. (2011). Minnesota Reading Corps Final Evaluation 2010-2011. Minneapolis, MN: MRC. 

8 The school that did not participate referenced staff shortages as the reason. 
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approach. 9 From this group of eligible PreK sites, a comparison group was identified by matching the site to a Minnesota 
Reading Corps PreK program site on a select group of educationally important pre-intervention characteristics 10 (i.e., urbanicity, 
institution type, classroom composition of student ages, percentage of students eligible for FRPL, * 11 and percentage of students 
who were DLLs). Additional secondary variables were also considered when matching program and comparisons sites (e.g., site 
enrollment, instructional time, and student to teacher ratio), though they were primarily used to sharpen the match when multiple 
potential match sites were identified. The final set of variables is provided in Table 1. 


Table 1 . PreK QED Matching Variables 


Order 

Variable 

Description 

Primary Matching Variables 

1 

Urbanicity 

Urban, suburban or rural 

2 

Institution Type 

Public preschool, Head Start Center, Community-based program 

3 

Classroom Composition of 
Student Ages Served 

Only 4/5 year olds or mixed 3,4 and 5 year olds 

4 

Socio-Economic Status (SES) 

% students eligible for Free or Reduce Price Lunch (FRPL) and/or % poverty of families 
with children in the census tract within which the site is located 

5 

Dual Language Learner (DLL) 
Status 

% of students who do not speak English as a first language. 

Secondary Matching Variables 

6 

Site Enrollment 

Number of students enrolled in the PreK program at the site. 

7 

Instructional Time 

Number of hours of instruction per week (e.g., half vs. full day; number of days per week) 

8 

Student to Teacher/Adult Ratio 

The number of students for every teacher or adult in the classroom. 


Once potential comparison sites were identified, the evaluation team approached site administrators to gain their participation in 
the study through a prescribed recruitment process. Sites typically agreed to participate because they would benefit from 
obtaining access to the assessment data we collected on their students upon completion of the study. In the end, the study team 
was able to match 29 schools to Minnesota Reading Corps programs and recruit 25 sites to participate in the study. Thus, 25 
pairs (25 program sites and 25 comparison sites) for a total of 50 sites participated in the outcome evaluation. The total student 
sample size was determined by the number of students enrolled in a PreK classroom. If a comparison site had more classrooms 
and students than was required to achieve sufficient statistical power (i.e., 30 students across two classrooms), the comparison 
site classrooms that best mirrored the matched-Minnesota Reading Corps program classroom on amount of instructional time 
(i.e., days per week, hours per day) were selected. 


9 Since SEEDS is an integral part of the Minnesota Reading Corps program, including SEEDS trained sites in the evaluation would have 
eliminated the possibility of detecting an effect of this critical program element. 

10 These variables were based on WWC recommendations, administrative data collected by Minnesota schools at the time, the literature on 
predictors of preschool emergent-literacy outcomes, and consultation with the project's Technical Working Group (TWG). 

11 A measure of household poverty and socio-economic status 
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Data Collection 

The evaluation team employed two methods of data collection for the PreK outcome evaluation: 1 ) administrative data retrieval 
from program sites and 2) original data collection from comparison sites. Because the Minnesota Reading Corps PreK program 
collects Fall, Winter, and Spring IGDI benchmark data from all PreK students in program classrooms, NORC requested access 
to all 2013-2014 school year assessment and demographic data from the 25 Minnesota Reading Corps PreK sites selected for 
participation in the outcome evaluation. This administrative data served as the primary outcome data for the program sites. 

NORC was responsible for collecting equivalent IGDI data three times a year (Fall, Winter, Spring) from PreK student at the 
comparison sites. In order to collect this data, the evaluation team sent field testers, who would be responsible for assessing 
students at comparison sites, to the same Minnesota Reading Corps training as that attended by AmeriCorps members. This 
process ensured that the field testers: 1) used the same IGDI data collection instruments for collecting student-level benchmarks 
from comparison site students as those used to collect data from students at the Minnesota Reading Corps program sites; and 2) 
received the same standardized training in assessment administration and scoring as the AmeriCorps members. This common 
use of instrumentation and training ensured commonality of assessment practices across both program and comparison sites. 

Analysis 

The student-level IGDI assessment and demographic data NORC collected from comparison sites were combined for analysis 
with de-identified IGDI data from the Minnesota Reading Corps administrative dataset. The resulting analytic dataset contained 
all the assessment and demographic data collected from comparison sites and existing data extracted for participating Minnesota 
Reading Corps PreK program sites. 

As discussed above, each Minnesota Reading Corps program site was paired with a comparison site on several key variables. 
During data analysis, these variables were equated using statistical controls so as to adjust for pre-intervention differences 
among groups. This produced a “site pair,” comprised of one Minnesota Reading Corps program site and one matched 
comparison site. The analysis was conducted separately for 3-year-old students and for 4- and 5-year-old students due to 
several factors: 1) 3-year-old children are developmental^ distinct from 4- and 5-year old children; 2) there is more variability in 
prior exposure to PreK instruction among 4- and 5-year old students than among 3-year old students; and 3) the assessments 
used to measure letter name fluency differed by age group. 

Three specific and separate analytic approaches were used to address the three major research questions of the evaluation: 

1. To assess the average treatment effect of the Minnesota Reading Corps PreK program (RQ1), the evaluation team 
examined differences within each site pair by each student age group by employing an ANCOVA (Analysis of 
Covariance) regression using fixed effects for site pairs. The ANCOVA regression predicted the Spring IGDI 
benchmark score (outcome) as a function of the student-level covariates, namely the Fall benchmark score and age 


OUTCOME EVALUATION OF THE MINNESOTA READING CORPS PREK PROGRAM 


Page x 


The Corporation for National and Community Service \ 2015 


(i.e., 3-year old and 4/5-year old), in addition to the site-level Minnesota Reading Corps PreK program participation 
variable (indicator). 12 

2. To determine whether the effect on emergent literacy outcomes varied as a function of student demographics (RQ2), 
the evaluation team employed two types of subgroup analyses. 13 First, specific covariates and moderators were added 
to the ANCOVA model and jointly tested to determine if there were significant differences in effects across major 
subgroups. The second analysis involved subdividing the data by major subgroup and applying the same analysis as 
for the entire sample. Subgroups were defined based on characteristics provided at baseline and each level was 
determined by individual characteristics, including gender, race/ethnicity, and Dual Language Learner (DLL) status. 

3. Given the diversity of early childhood education programs (e.g., curricula, attendance time, student/family 
demographics, funding source, etc.), it was unclear if and how effective the Minnesota Reading Corps PreK program 
might be at different sites and site types. To estimate whether program effects varied by site (RQ3), the evaluation 
team conducted two analyses: 1) a subgroup analysis on site type (i.e., public preschool or Head Start Center 14 ) and 2) 
an intraclass correlation (ICC) analysis to assess site pair-level effects. The site type subgroup analysis estimated the 
differential impact of the Minnesota Reading Corps PreK program in public preschools and Head Start centers. The 
ICC analysis examined whether there was variation in the effect of the program across pairs of sites (program vs. 
comparison). 

Findings and Conclusions 


Below, the evaluation team offers our conclusions based on the study findings and organizes them by the three major research 
questions, followed by final thoughts on the implications of these findings for the future of the Minnesota Reading Corps PreK 
program. 

Research Question #1: What is the effect of the Minnesota Reading Corps program on student literacy outcomes? 

The results of our analysis showed that preschool students who attended Minnesota Reading Corps PreK classrooms achieved 
significantly higher emergent literacy assessment scores by the end of the school year than did students who did not attend a 
Minnesota Reading Corps classroom. The Minnesota Reading Corps PreK program had statistically significant effects across all 
five IGDI outcome measures, letter sound fluency, rhyming fluency, letter name fluency, picture name fluency ,and alliteration 
fluency, for 4- and 5-year old students between Fall benchmark (September 2013) and Spring benchmark (May 2014) . In 
addition, medium to large effect sizes were found on each measure, ranging from .40 to .72, indicating that the program has a 
comprehensive, positive effect on students’ emergent literacy skills. Despite the substantially smaller sample size for 3-year old 


12 The analysis team also conducted an analysis of the Winter IGDI benchmark scores, and the results are available in Appendix B to this 
report. 

13 Subgroup analyses were only conducted on the 4- and 5-year old student sample. They were not conducted on the 3-year old student 
sample due to small sample sizes. 

14 Head Start Centers restrict enrollment to low-income households (i.e., at or below federal poverty level for household size). Thus, enrollment 
in a Head Start Center is a proxy for lower socio-economic status (SES). 
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students, the Minnesota Reading Corps PreK program was found to have a significant and medium to large effects on 3-year old 
PreK students’ rhyming fluency and picture name fluency scores between Fall and Spring benchmark. Although growth was also 
found for 3-year old students on the letter name recognition and alliteration fluency outcome measures, 3-year old PreK students 
in Minnesota Reading Corps PreK classrooms did not attain significantly higher scores on these measures at Spring benchmark 
than PreK students in comparison classrooms. 


Figure 2. Example of differential growth in 4- and 5-year old PreK program and comparison students’ emergent literacy 
outcomes: Letter Sound Fluency 



Overall, the results of the student outcome analysis revealed that the Minnesota Reading Corps PreK Program is achieving its 
goal of creating Kindergarten-ready 4- and 5-year old students, as measured against the IGDI Spring benchmarks. Of particular 
interest was the large, significant effect that the Minnesota Reading Corps program produced in the growth in children's 
vocabulary, as measured by picture name fluency. Vocabulary is one of the strongest predictors of later reading comprehension. 
The Minnesota Reading Corps PreK program’s attention to not only decoding skills (i.e., letter name recognition and letter sound 
correspondence) but also oral language skills (i.e., vocabulary) is a key program feature and likely a primary reason for the 
demonstrated effectiveness of the program. The contextualized language and literacy instruction emphasized by the program, as 


OUTCOME EVALUATION OF THE MINNESOTA READING CORPS PREK PROGRAM 


Page xii 


The Corporation for National and Community Service \ 2015 


best evidenced by extended adult-child discourse and rich dialogic reading sessions, is a proven method for growing children's 
emergent literacy skills. 

Research Question #2: Does the effect on literacy outcomes vary by student characteristics/demographics? 

For the most part, students from all subgroups examined produced significant growth in emergent literacy outcomes regardless 
of gender, race/ethnicity, or DLL status. Of note, students typically at risk for not being Kindergarten-ready, non-Whites and Dual 
Language Learners, posted significantly greater growth in vocabulary, as measured by picture name fluency, at Minnesota 
Reading Corps PreK sites than at comparison sites. In contrast to students at program sites, the vocabulary of non-White 
students and DLL students at comparisons sites did not grow significantly from Fall to Spring. This finding implies that the 
program elements that together comprise the Minnesota Reading Corps PreK program have a significant and meaningful impact 
on a critical skill (vocabulary) at program sites that is otherwise unaffected at comparison sites. 

Research Question #3: Do effects on literacy outcomes vary by site/school? 

The Minnesota Reading Corps program appears to be consistently effective no matter the type of preschool setting in which the 
program is implemented, either public school or Head Start center. Students enrolled at both types of preschools evidenced 
greater growth in emergent literacy outcomes at Minnesota Reading Corps PreK program sites than comparison sites. Of note, 
students at Minnesota Reading Corps Head Start centers achieved significantly greater growth on letter sound fluency and 
alliteration fluency than their peers at comparison Head Start centers who did not improve on these measures during the school 
year. These results suggest that not only is the Minnesota Reading Corps PreK program effective in both public schools and 
Head Start centers, but that the program may be particularly effective in developing some emergent literacy skills for students 
enrolled in Head Start centers. 

Final Thoughts 


Overall, the results of this study show that the Minnesota Reading Corps PreK program is an effective model for improving 
preschool students' emergent literacy skills. Thus, the Minnesota Reading Corps PreK program is contributing to the nationwide 
goal of improving 3 rd grade reading proficiency by preparing preschool-aged students to be kindergarten-ready. 

The Minnesota Reading Corps PreK program’s comprehensive scope, including both whole-class instructional strategies and 
differentiated interventions, results in class-wide impacts. The coordination between the two forms of instruction is likely a key 
source of the accelerated growth we observed in students’ emergent literacy outcomes. As outlined in the Phase II Process 
Assessment, 15 the Minnesota Reading Corps PreK program provides a common infrastructure for coordinated, intentional 
instruction through the Literacy Rich Schedule, a common pedagogy for interacting with PreK students via SEEDS, and proven 
strategies for embedding language and literacy instruction at times during the school day that are often overlooked (e.g., arriving 
at the beginning of the day, lining up for bathroom breaks or the bus, transitioning between circle time and choice time). 


15 Hafford, C., Markovitz, C., Hernandez, M.W., Diaconis, A., Estrera, E., & Muyskens, P. (2014). Phase II Process Assessment of the 
Minnesota Reading Corps PreK Program. (Prepared under contract to the Corporation for National and Community Service). Chicago, IL: 
NORC at the University of Chicago. 
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Furthermore, the program is most effective when all adults in a classroom coordinate their practice; therefore, it is critical that the 
lead teacher understand and fully commit to the Minnesota Reading Corps PreK model. The Minnesota Reading Corps 
program’s use of a multi-layered supervisor structure, including site-based Internal Coaches and program-provided Master 
Coaches, effectively supports the implementation of such a comprehensive program. In sum, the complete Minnesota Reading 
Corps PreK program, implemented with fidelity, is both highly replicable and effective in improving diverse PreK students' 
emergent literacy outcomes. 
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I. Introduction 

Minnesota Reading Corps is a statewide initiative with a mission to help every Minnesota child become a proficient reader by the 
end of third grade. The Minnesota Reading Corps program engages a diverse group of AmeriCorps members to provide literacy 
enrichment and tutoring services to preschool children (PreK) and at-risk Kindergarten through third grade (K-3) elementary 
school students. As of the 2013-2014 school year, more than 1,100 AmeriCorps members 16 implemented the program in 712 
schools or sites 17 and 213 school districts across the state of Minnesota. 18 

This report, funded by the Corporation for National and Community Service (CNCS), describes the findings of a quasi- 
experimental design (QED) outcome evaluation of 1,534 three-, four-, and five-year old students who attended either a 
Minnesota Reading Corps PreK program sites or a comparison site during the 2013-2014 school year. Students were enrolled at 
a representative sample of 25 Minnesota Reading Corps programs and 25 comparison sites matched to each Minnesota 
Reading Corps preschool or Head Start program on specific site-level characteristics. The goal of the outcome evaluation was to 
determine the effects of the Minnesota Reading Corps program on preschool students’ emergent literacy skills. 19 

The PreK outcome evaluation is one of several complementary studies of the Minnesota Reading Corps program: a process 
assessment of the Minnesota Reading Corps program in 20 PreK and K-3 sites (completed in Spring 201 3) 20 a randomized 
controlled trial (RCT) impact evaluation of the Minnesota Reading Corps K-3 rd program (Spring 2014) 21 and a survey of 
AmeriCorps members (Fall 2013). In addition to these studies, a Phase II process assessment focusing specifically on the 
Minnesota Reading Corps PreK program was completed in Fall 2014, the results of which were intended to inform the findings 
presented in this report. 

During a feasibility study for designing the Minnesota Reading Corps evaluation, we concluded that the random assignment of 
individual PreK students to program and comparison groups within a school was not feasible. The Minnesota Reading Corps 
PreK program is designed to serve an entire classroom of students and transform teachers' instructional practices, which affects 
all students. As such, it was not possible to randomly assign either students within classrooms or classrooms within schools to 


16 324 members served in the PreK program during the 2013-2014 school year; 5,174 Minnesota Reading Corps members have served from 
2003 through 2014. 

17 According to the Minnesota Department of Education (MDE), during the 2013-2014 school year, 1618 public schools served grades K-12. Of 
those schools, 933 offered PreK services. The total number of preschools in the state of Minnesota (i.e., public schools and non-public schools) 
was not available. http://w20.education.state.mn.us/MDEAnalvtics/Summarv.isp 

18 According to MDE, during the 2013-2014 school year, there were 328 public operating elementary & secondary independent school districts, 

3 intermediate school districts, and 150 charter schools (which are considered public school districts in Minnesota). 

19 Specifically, letter sound fluency and letter name recognition or fluency (alphabet knowledge), rhyming and alliteration fluency (phonological 
awareness), and picture name fluency (vocabulary) 

20 Hafford, C., Markovitz, C., Hernandez, M, et al. (February 2013). Process Assessment of the Minnesota Reading Corps Program. (Prepared 
under contract to the Corporation for National and Community Service). Chicago, IL: NORC at the University of Chicago. 

21 C., Markovitz, C., Hernandez, M, et al. (March 2014). Impact Evaluation of the Minnesota Reading Corps K-3 Program. (Prepared under 
contract to the Corporation for National and Community Service). Chicago, IL: NORC at the University of Chicago. 
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program and comparison conditions. 22 Therefore, the best alternative, a rigorous QED, was employed. The Minnesota Reading 
Corps PreK outcome evaluation focuses on the following research questions: 

1 . What is the effect of the Minnesota Reading Corps program on student literacy outcomes? 

2. Does the effect on literacy outcomes vary by student characteristics/demographics? 

3. Do effects on literacy outcomes vary by site/school? 

To address these questions, we begin in Chapter II by presenting a brief overview of the Minnesota Reading Corps program and 
its role in the recruitment, training, placement and monitoring of AmeriCorps members as they implement the program in 
preschool and elementary school settings. We then describe the PreK component of the Minnesota Reading Corps program 
(which is the focus of this evaluation), its multi-layered supervisory structure, and the Summer Training Institute. Chapter III then 
provides information on the outcome evaluation's methodology for selecting sites to participate in the study, procedures for 
matching Minnesota Reading Corps (program) and comparison sites, data collection and use of program data, and analysis of 
findings. 

This background information sets the context for the presentation of findings from our analyses of assessment data from the 
Minnesota Reading Corps PreK program in Chapter IV. These findings are based on a comparison of Spring outcome scores for 
students attending a Minnesota Reading Corps program site and similar students enrolled in a comparison program site. Our 
examination of the program outcomes also includes analyses of key subgroups, including gender, race/ethnicity, and Dual 
Language Learner (DLL) status. We did not examine students by Free or Reduced Price Lunch (FRPL) eligibility, which is a 
measure of socio-economic status, because these data were either not collected or considered confidential; however, the 
majority of our sample attended either a Head Start program (100% FRPL eligible) or public schools with high percentages of 
elementary school-aged children with who were FRPL eligible. Therefore, it is reasonable to assume that the majority of the 
children in our study would likely meet eligibility criteria for FRPL. In this chapter, we also examine whether outcomes vary due to 
the type of site where the intervention took place (i.e., public school or Head Start center). 

In addition to the presentation of findings on the study results in Chapter IV, we provide in Appendix D a separate analysis of the 
data tailored to the requirements of the U.S. Department of Education's Institute of Education Sciences' What Works 
Clearinghouse (WWC). We share the WWC's goal of providing educators with the information they need to make evidence- 
based decisions. Therefore, we have developed this appendix to specifically demonstrate that our study meets WWC's 
standards for a QED study that “meets evidence standards with reservations.” 

We conclude our report in Chapter V by returning to the research questions. The evaluation team addresses whether the 
Minnesota Reading Corps PreK program appears to have an effect on students’ emergent literacy skills and whether there are 
differential effects by age, gender, race/ethnicity, and/or DLL status based on growth between Fall and Spring benchmark on 


22 In addition to randomizing students or classrooms, another option which was considered was to randomize schools or sites. However, there 
are not enough new school applicants to the Minnesota Reading Corps program each year to be able to adequately power an impact 
evaluation where the unit of randomization is schools/sites. 
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age-appropriate literacy measures. Finally, we discuss the implications of the findings for the Minnesota Reading Corps program. 
A glossary of terms to assist the reader is provided in Appendix E. 
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II. About Minnesota Reading Corps 

A. Statewide Implementation of the Minnesota Reading Corps: 2003-2013 

Minnesota Reading Corps is the largest AmeriCorps State program in the country. The goal of the Minnesota Reading Corps is 
to ensure that students become successful readers and meet reading proficiency targets by the end of the third grade. The 
Minnesota Reading Corps program was started in 2003 to provide reading and literacy tutoring to children in four preschool 
(PreK) Head Start programs. In 2005, the Minnesota Reading Corps expanded its program to serve students in Kindergarten 
through third grade (K-3). The core activities of the Minnesota Reading Corps, and its host organization, ServeMinnesota Action 
Network, are to recruit, train, place and monitor AmeriCorps members to implement research-based literacy interventions for at- 
risk preschool children and K-3 students. 

The Minnesota Reading Corps is a strategic initiative of ServeMinnesota. ServeMinnesota is the state commission for all 
AmeriCorps State programs in Minnesota, including the Minnesota Reading Corps, and helps leverage federal, state and private 
dollars to operate the Minnesota Reading Corps. As a catalyst for positive social change and community service, 

ServeMinnesota works with AmeriCorps members and community partners to meet critical needs in Minnesota. As a nonprofit 
organization, it supports thousands of individuals to improve the lives of Minnesotans by offering life-changing service 
opportunities that focus on education, affordable housing, employment, and the environment. The ServeMinnesota Action 
Network serves as fiscal host to provide statewide management and oversight for the Minnesota Reading Corps program. The 
Action Network is a nonprofit organization and serves as a home to incubate, replicate and scale evidence-based AmeriCorps 
programs that address critical state priorities. In addition, the Saint Croix River Education District (SCRED) and TIES 23 have 
been funded by ServeMinnesota to conduct an annual evaluation of the Minnesota Reading Corps program. 24 

AmeriCorps members in the Minnesota Reading Corps program serve in school-based settings to implement Minnesota Reading 
Corps literacy enrichment strategies and conduct interventions with PreK-3 students. Minnesota Reading Corps members serve 
as AmeriCorps members, bound to the program’s call to service. As a direct service program, the Minnesota Reading Corps 
engages its members in service to work towards the solution of a social issue. In exchange for their service of 1 ,700 hours a year 
(full-time) or 900 hours a year (part-time), members receive benefits that include a bi-weekly stipend, student loan forbearance, 
and an education stipend for the first two years of service. 

Minnesota Reading Corps AmeriCorps members are supported by a multi-layered supervisory structure. One or more on-site 
Internal Coaches mentor members during their year of service, continually monitoring fidelity of program implementation, and 
ensuring effective tutoring. Internal Coaches are typically specialists, teachers, or curriculum directors employed by the site or 
school. Expert-level Master Coaches are assigned to each Internal Coach to provide consultation on literacy interventions and 
assessment, as well as ensure fidelity to the Minnesota Reading Corps model. The Minnesota Reading Corps Program 


23 http://ties.k12.mn.us/ 

24 ServeMinnesota 201 1 , Background document. 
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Coordinators provide administrative support to individual sites (Principals, Internal Coaches, and Master Coaches) and assist 
members with their AmeriCorps responsibilities. 

In the 2013-14 school year, the Minnesota Reading Corps program’s more than 1,100 AmeriCorps members served over 30,000 
students in 712 elementary schools, Head Start centers, and preschools, making it the largest AmeriCorps programs in the 
country. Based on the early success of the Minnesota Reading Corps program, replication is underway in Colorado, 
Massachusetts, Michigan, Santa Cruz County, CA, Washington, DC, Virginia, Iowa, and North Dakota. 

B. Foundational Framework and Staffing Structure in the Minnesota Reading Corps 

The Minnesota Reading Corps program utilizes a Response to Intervention (Rtl) approach to objectively identify and provide 
additional instructional support to at-risk students. Rtl is based on a problem solving approach, which was incorporated into the 
2004 Individuals with Disabilities Education Act (IDEA), and has been gaining popularity among educators, policymakers, 
administrators, teachers, and researchers. The key features of the Minnesota Reading Corps Rtl framework are: 

■ Clear literacy targets at each age level from PreK through grade 3 

■ Benchmark assessment three times a year to identify students eligible for one-to-one and small group interventions 

■ Scientifically based interventions 

■ Frequent progress monitoring during intervention delivery 

■ High-quality training and coaching in program components, literacy assessment and instruction 

In Rtl, assessment data play the key roles of screening students’ eligibility for additional services and then monitoring students’ 
progress towards achieving academic goals (i.e., benchmarks). The Minnesota Reading Corps screens students for program 
eligibility three times a year (i.e., Fall, Winter, Spring) with two sets of grade-specific, literacy-focused general outcome measures 
(i.e., IGD1 1.0 for PreK and FAST for K-3) that possess criterion-referenced grade- and content-specific performance 
benchmarks. Program staff use scores from these general outcome measures to categorize students into one of three possible 
tiers (i.e., proficiency levels; see Figure 11.1): Tier 1 students score at or above benchmark and benefit from typical whole class 
instruction (75-80% of students score in this category); Tier 2 students score below benchmark and require supplemental 
intervention until they meet benchmarks (15-20% of students fall into this category); and Tier 3 students require intensive one-to- 
one intervention provided by a special education teacher or literacy specialist and often have individualized educational plans (5- 
10% of students qualify for this category). 25 


25 The percentage of children testing into Tiers 1-3 are national estimates based on the literature: 
http://www.rtinetwork.org/essential/tieredinstruction/tiered-instruction-and-intervention-rti- 

model ; http://www.hanoverresearch.com/insiqhts/developinq-a-maximuiTi-impact-response-to-intervention-proqram/7Fk-1 2-education : Hoover, 
J.J., & Patton, J.R. (2008). The role of special educator in a multitiered instructional system. Intervention in School and Clinic , 43(4), 195-202. 
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Figure 11.1. Minnesota Reading Corps Response to Intervention Tiers 



TIER 1 

Whole class, high-quality instruction (Pre-K) 
75%-80% 


The Minnesota Reading Corps PreK program includes both an immersive “push-in” component, where members provide whole- 
class literacy enrichment for all students (i.e., Tier 1), as well as targeted small group and one-to-one components, where 
members provide more individualized interventions to students struggling with emergent literacy skills (i.e., Tiers 2 and 3). The 
Minnesota Reading Corps K-3 program provides one-to-one tutoring where members provide supplemental individualized 
literacy interventions to primarily Tier 2 students in Kindergarten through third grade. Although the Minnesota Reading Corps 
program provides both PreK and K-3 interventions to students, the focus of this evaluation is on the Minnesota Reading Corps 
PreK program. Therefore, the remainder of this report will focus on describing the PreK program and evaluation. As previously 
mentioned, the findings from a randomized controlled trial (RCT) evaluation of the Minnesota Reading Corps K-3 program were 
published in Spring 2014. 

Overview of PreK Program Literacy Focus and AmeriCorps Members’ Role 

At the PreK level, the Minnesota Reading Corps program focuses on integrating the “Big Five” Early Literacy Predictors outlined 
by the National Reading Panel 26 into all aspects of the daily classroom routine. The “Big Five” for preschool students include: 1) 
conversational skills, 2) vocabulary and background knowledge, 3) book and print rules, 4) phonological awareness (i.e., rhyming 
and alliteration), and 5) alphabetic knowledge (i.e., letter name recognition and letter sound correspondence). AmeriCorps 
members are tasked with creating a Literacy Rich Classroom using evidence-based practices assessed in the ELLCO (Early 
Language & Literacy Classroom Observation) tool, and implementing a Literacy Rich Schedule. Along with implementing 
classroom-based strategies, members provide targeted individual or small group literacy tutoring for Tier 2 and Tier 3 students 


26 National Early Literacy Panel. (2008). Developing early literacy: Report of the National Early Literacy Panel. Washington, DC: National 
Institute for Literacy. 
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until they meet program-specified targets that predict end-of-year proficiency. Members implement 5-to 10-minute scripted 
interventions focused on students’ emergent literacy skills. 

The Minnesota Reading Corps Literacy Rich Schedule is designed to provide children with daily routines that embed early 
literacy predictors into fun and meaningful learning. As such, each classroom’s day is structured around a Literacy Rich 
Schedule that is integrated with the school’s curriculum. The schedule includes 12 activities, in order: Arrival, Sign-in, Meal Time, 
Large Group, Daily Message, Repeated Read Aloud, Tier 1 Small Group, Journal (weekly), Choice Time/Active Learning, Tier 2 
or Tier 3 Small Group, Big 5 transitions, and Family (through Talk, Read, and Write with Mel). Within and between each 
scheduled activity, teachers and members strive to integrate Minnesota Reading Corps expected routines, including “Strive for 5” 
conversations using an overarching theme, functional vocabulary, and “Big 5 Transitions.” The Big 5 Transitions occur as 
children move from one part of the literacy rich schedule to another. Members and teachers engage the children in an activity, 
such as rhyming games or letter sound songs, focused on one of four emergent literacy skills: oral language, phonological 
awareness, letter names, and letter sounds. 

AmeriCorps members serving in the PreK program are also responsible for enacting the SEEDS of Emergent Literacy, 27 which 
provides both members and teachers with specific strategies to enhance literacy instruction for all children in the classroom. The 
SEEDS model is interactive, skills-focused and based on current research in early childhood education, child development, 
emergent literacy, and effective teaching. SEEDS is a relationship-based instructional approach that maps out for teachers five 
ways to Intentionally interact with children in order to promote academic growth and social-emotional well-being. SEEDS high 
quality interactions include the following five elements: 

■ Sensitivity: Look, listen, and ask questions to become aware of each child's needs, thoughts, abilities and feelings; 

■ Encouragement: Use intentional affirmations and positive non-verbal communication to create a shared positive learning 
environment; 

■ Education: Embed the “Big 5” literacy skills in daily routines (vocabulary, conversation, phonological awareness, book and 
print rules, and letter knowledge); 

■ Development of Skills Through Doing: Help children explore their world through hands-on learning; and 

■ Self-Image Support: Balance the SEEDS quality interactions to support a child’s feeling of being respected and capable. 

Within the PreK program, there are different roles for instructional staff and AmeriCorps members. Each PreK classroom has a 
“lead teacher” (a licensed teacher who is the students’ primary instructor) and an AmeriCorps member. Two types of AmeriCorps 
members may be assigned to a classroom: Educator Corps 28 or Community Corps. Community Corps members are recruited 
from the community and are embedded into a classroom to collaborate with teaching staff. Educator Corps members are current 
employees of the service site who receive additional training and accept new responsibilities to become a non-stipend 
AmeriCorps members. Both Community Corps and Educator Corps members’ share the same goal of developing their students’ 


27 Horst, K„ & Passe, A. (2004). Creatine Literacy Rich Classrooms for Preschool Children (Ages 0-5). Presented at the 2004 CEED 
Symposium. Minneapolis, MN. 

28 In previous years, Educator Corps members were referred to as Professional Corps members. 
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emergent literacy skills in preparation for Kindergarten. Their role in the classroom is to provide children with high quality 
language and literacy learning opportunities by completing assessments, implementing the literacy rich schedule, engaging in 
SEEDS quality interactions, and conducting Tier 2 and 3 small-group and one-to-one interventions. 

Supervisory Staff 

The Internal Coaches and Master Coaches play important roles in Minnesota Reading Corps program implementation (see 
Figure 11.2 for an illustration of the complete Minnesota Reading Corps supervisory structure). The Internal Coach is a school 
employee who is trained to provide on-site literacy support and oversight to AmeriCorps members serving as literacy tutors at the 
site. In order to ensure fidelity to the Minnesota Reading Corps model, the Internal Coach conducts integrity observations of 
members as they assess students (three times per year) and implement interventions with students (once a month). The Internal 
Coach provides the member with feedback for continual improvement based on these observations. The Internal Coach also 
ensures that the member is accurately reporting student data in edSpring, an internet-based data entry system for storing 
progress monitoring data on students. Throughout the school year, the Internal Coach works with assistance from the Master 
Coach to select appropriate interventions for each student and to determine if students are ready to exit from Tier 2 or 3 
intervention. The Master and/or Internal Coaches also complete an Early Language and Literacy Classroom Observation 
(ELLCO) in the Fall and Spring to provide objective feedback and suggestions to the teaching team to improve the classroom’s 
language and literacy environment. The Internal Coach works closely with Minnesota Reading Corps program staff and school 
administration to address any concerns about member performance and to conduct disciplinary action if necessary. The 
Minnesota Reading Corps estimates that the time commitment for Internal Coaches is 6-9 hours per member per month. The 
additional time commitment for required training is 56 hours for new PreK Internal Coaches and 24 hours for returning PreK 
Internal Coaches. 29 

Figure 11.2. Minnesota Reading Corps Supervisory Structure 
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The Master Coach is a literacy expert employed by the Minnesota Reading Corps who serves as a literacy consultant to the 
Internal Coach and member(s). The Master Coach supports the Internal Coach and the member in making decisions about 


29 http://minnesotareadinqcorps.orq/sites/default/files/PreK%20Guide%20for%20Sites%2014-15.pdf 
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student eligibility for Tier 2 or 3 interventions by reviewing benchmark IGDI data. The Master Coach also helps to ensure fidelity 
to the Minnesota Reading Corps model. The Master Coach visits schools at different frequencies throughout the year depending 
on the schools’ degree of experience implementing the Minnesota Reading Corps program, ranging from once a month for 
schools that have recently implemented the program to three times a year for schools where the program is well-established. 
Visits last approximately one hour per member, during which the Master Coach, Internal Coach and member(s) examine student 
assessment data, discuss interventions choices, review student progress, and discuss implementation challenges. 

Other Master Coach responsibilities include communicating with the Internal Coach and member(s) about preparing for 
benchmarking; performing member fidelity checks along with the Internal Coach to ensure appropriate administration of 
benchmark assessments and interventions; providing consultation as needed regarding the identification and prioritization of 
students to receive small group or one-to-one interventions; reviewing student progress; and providing program updates to the 
Internal Coach and member. If the Internal Coach cannot answer a member’s question, the Master Coach can often provide 
advice. The Master Coach can also answer questions about topics such as data entry or scheduling. 

For administrative issues, such as questions about training schedules and timesheets, the Internal Coach or member can contact 
their Minnesota Reading Corps Program Coordinator. The Program Coordinator also helps members answer questions about 
their community service requirement and requested leaves of absence. Program Coordinators are also notified about all member 
disciplinary issues. 

C. Summer Institute Training 

Each summer, the Minnesota Reading Corps hosts a multi-day Summer Institute for training returning and new Master Coaches, 
Internal Coaches, and AmeriCorps members. 30 ServeMinnesota and Minnesota Reading Corps staff orchestrates the 
organizational and administrative aspects of the Summer Institute, while literacy experts conduct training sessions. This 
intensive, information-filled conference provides expert training in the evidence-based literacy interventions employed by the 
Minnesota Reading Corps. In its most basic form, the Summer Institute is a learning forum for literacy interventions and teaching 
techniques. However, the Summer Institute also serves an important role in developing member, coach, and school adherence 
to the Minnesota Reading Corps model. Speeches from former and current members, funders, parents, and officials from the 
Minnesota Department of Education and local school districts encourage this process and enhance the inspirational atmosphere 
of the training sessions. At the Summer Institute, the members also meet with their Internal Coach, and sometimes Master 
Coach, with whom they will be working throughout the upcoming school year. 

During several intensive sessions at the Summer Institute, members learn the essential skills, knowledge, and tools needed to 
serve as effective literacy tutors. These sessions introduce members to the Minnesota Reading Corps PreK program model, the 
Literacy Rich Schedule, the SEEDS approach to high quality adult-child interaction, the interventions used in the Tier 2 and 3 
small group and one-to one tutoring sessions, as well as the underlying research and theories supporting the interventions and 


30 Members attend all four days of the Summer Institute (one day orientation and three days of training). New Coaches attend three days, and 
returning Coaches attend one day. 
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program model. Importantly, members are provided with a detailed Literacy Handbook to serve as a constant resource for 
supporting program implementation. The Handbook provides an introduction to the Minnesota Reading Corps program, 
information on policies and procedures and service requirements, procedures for the benchmarking and progress monitoring of 
students, and specific direction and materials for implementing the Literacy Rich Schedule and conducting small group and one- 
to-one interventions. In addition, members are provided with online resources that mirror the contents of the Literacy Handbook 
and supplement it with other resources such as exemplar photos of a Literacy Rich Classroom, and videos of model 
interventions and best practices. Both the Handbook and website are intended to provide members with just-in-time support once 
they are out in their service sites, as well as with opportunities for continued professional development and skill refinement. 

At the Summer Institute, PreK members receive training on the SEEDS of Emergent Literacy approach, which serves as the 
pedagogical framework within which members and teachers create a literacy rich classroom environment. In addition to the 
Summer Institute, PreK members participate in multiple smaller trainings throughout the school year. These sessions include 
additional training on SEEDS, instruction in administering and scoring student assessments, and implementation of large and 
small group interventions. The trainings are generally one-day long and are held in multiple regional locations throughout the 
state. Each PreK site's Internal Coach is expected to attend the sessions with their member(s). 

The Summer Institute is the primary training venue not only for members, but also for Internal Coaches. At the Summer Institute, 
Internal Coaches receive a comprehensive orientation to the Minnesota Reading Corps PreK program, including program and 
early literacy background, intervention delivery, benchmarking, and progress monitoring. Internal Coaches also learn about their 
roles and responsibilities, including ensuring fidelity to the Minnesota Reading Corps model, orienting the member to the school, 
introducing school staff to the member, and coordinating school-based professional development opportunities for their 
members. Internal Coaches also are oriented to the layers of support provided by the Minnesota Reading Corps program, 
including the Master Coach and Program Coordinator. 

D. The Role of Assessment Data in the Minnesota Reading Corps Program Implementation and 
Improvement 

In the Minnesota Reading Corps PreK program, AmeriCorps members collect general outcome measure data on all PreK 
students in a classroom in the Fall, Winter, and Spring of each school year using the age-appropriate Individual Growth and 
Development Indicators (IGDIs) assessments. Internal Coaches review this data to then objectively determine eligibility for Tier 2 
or 3 interventions based upon their benchmark score. Members collect monthly progress monitoring data on those students 
selected to receive small group or one-to-one interventions. The IGDIs assess the following five critical emergent literacy skills: 

1 . Letters Sound Fluency (LSF) assesses how many lowercase letter sounds a student can identify and say (e.g., I, f, 
m) within one minute. Students receive one point for every correct letter sound provided within one minute. The total 
number of correct letter sounds is the Letter Sound Fluency score. Letter sounds fluency is considered a relatively 
difficult alphabetic knowledge skill and is therefore only assessed among 4- and 5-year old students, not 3-year old 
students. 
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2. Rhyming Fluency (RF) tests students’ phonological awareness by assessing how many rhyming words a student can 
correctly identify (e.g., bat and cat, car and jar, ham and jam) within two minutes. Students receive one point for every 
correct rhyme identified within two minutes. The total number of correct rhymes is the Rhyming Fluency score. 

3. Letter Name Recognition (LNR) is measured using two different methods depending upon the age of the child. For 3- 
year old students, children are shown upper case letters and children are asked to name the letter. Students receive 
one point for every letter named correctly. The total number of letters named correctly is the Letter Name Recognition 
score. This test is untimed and there is a limited number of letters, thus the 3 year old letter name test is not a fluency 
test. In contrast, 4- and 5-year old students complete a Letter Name Fluency (LNF) test designed to assess how 
many upper case letter names a student can identify within one minute. Students receive one point for every letter 
named correctly within one minute. The total number of letters named correctly is the Letter Name Fluency score. Both 
LNR and LNF assess students’ alphabel knowledge. Given the difference in testing methods, however, the results of 
the 3- and 4/5-year old letter name assessments cannot be directly compared. 

4. Picture Name Fluency (PNF) tests students' vocabulary by assessing how many pictures a child can name correctly 
within one minute. Students receive one point for every picture named correctly within one minute. The total number of 
pictures named correctly is the Picture Name Fluency score. 

5. Alliteration Fluency (AF) tests students’ phonological awareness by assessing how many alliteration matches (e.g., 
hill and heart, cat and can, bike and ball) a child can correctly identify within two minutes. Students receive one point 
for every correct alliteration identified within two minutes. The total number of correct alliterations is the Alliteration 
Fluency score 

Students’ scores on the IGDIs are used to identify PreK students who are eligible for Tier 2 or 3 interventions. Members then 
collect monthly progress monitoring data on those Tier 2 and 3 students to whom they provide individualized or small-group 
interventions. Table 11.1 lists the specific IGDI assessments and corresponding benchmark scores used to identify eligible 4- and 
5-year old PreK students for Tier 2 and 3 interventions. The benchmarks for Winter and Spring are the same for 4- and 5-year 
old PreK students. These benchmark scores correspond to empirically-derived target scores based on research from the 
Minneapolis Public Schools and the St. Croix River Education District, who both have a long history of implementing the IGDI 
assessments at scale and with fidelity (Bollman, 2003). 31 The benchmark scores indicate the level of literacy skill that needs to 
be demonstrated in order to be considered “on-track” to developing literacy skills, which in turn is an indicator of high likelihood of 
reaching grade level standard on a high-stakes state reading assessment in third grade. For example, in order for a 4-year old to 
be “on-track” to be a successful reader by third grade, they need to have a score of at least 26 on the Spring picture name 


31 Bollman, K. (2003). Analysis of IGDI performance in Minneapolis Public Schools and St. Croix River Education District, for 
target setting with 4- and 5-year old students. [Technical Data Analysis] Rush City, MN. 
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fluency assessment. Research to set benchmarks for IGDIs for 3-year old students is more limited, so we did not use benchmark 
scores for the 3-year old students participating in our analysis. 32 


Table 11.1. Minnesota Reading Corps PreK Assessments and Winter/Spring Targets for 4- and 5-Year Old Students 


Fluency Assessment 

Winter/Spring Target 

Letter Sounds 

10 

Rhyming 

12 

Letter Names 

14 

Picture Names 

26 

Alliteration 

8 


*The research to set benchmark targets for IGDIs for 3-year-old students is more limited and, therefore, we only provide benchmarks for 4- and 5-year-old 
students. 


The Minnesota Reading Corps program uses the edSpring internet-based data entry system to record and store IGDI benchmark 
and progress monitoring data on all students served by the program. Monthly progress monitoring allows members to chart 
growth in emergent literacy outcomes among students receiving Tier 2 or 3 interventions. Progress monitoring data enables 
members to evaluate the effectiveness of their interventions, gauge whether students might benefit from a change in 
interventions, or determine whether they are ready to exit from intensive intervention. Every student's progress monitoring scores 
are graphed and then reviewed monthly by a collaborative team consisting of the member, Internal Coach and Master Coach. 

The Master Coach, Internal Coach, and AmeriCorps member discuss each student's assessment results over time before 
deciding to exit the student from Tier 2 or 3 tutoring. 

The data intensive orientation of the Minnesota Reading Corps program provides members, coaches, teachers and 
principals/directors with a consistent, objective means of identifying students who would benefit from intensive intervention, 
tracking their progress toward achieving academic goals related to critical emergent literacy skills, and informing instruction. The 
assessment data play an important role in garnering site-wide support from site staff, particularly as they see quantitative 
improvement in student outcomes. The data also provide members and coaches with objective information about the efficacy of 
the interventions with individual students, which can in turn be used to tailor the most effective instruction for the student's skill 
level. 


32 Due to these limitations on available benchmarks, 3-year old students are sometimes identified for Tier 2 and 3 interventions using 4-year old 
benchmarks. 
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Outcome Evaluation of the Minnesota Reading Corps PreK Program 

Building on the Minnesota Reading Corps program background provided in Chapter II, this chapter provides a detailed 
description of the methodology used to implement the Minnesota Reading Corps PreK outcome evaluation. The methodology 
was informed by the 2008-2009 and 2010-201 1 annual Minnesota Reading Corps evaluations. 33 The chapter begins with the 
presentation of a logic model for the Minnesota Reading Corps program outlining key program and school inputs and activities, 
as well as the program's desired short-term outcomes and long-term goals. After establishing the key components of the 
Minnesota Reading Corps PreK program, the evaluation team presents the three primary research questions for assessing the 
program’s effect on PreK students' emergent literacy skills (i.e., the logic model’s PreK short-term outcome). These research 
questions guided all aspects of the evaluation design, study implementation, and data analysis. 

Following our presentation of the research questions, the next section in this chapter describes the process for sampling 
Minnesota Reading Corps sites for the PreK evaluation. Next, the quasi-experimental design and site-level matching procedures 
for the study are provided, which includes key characteristics/demographics of the participating schools. The section also 
presents the pre-intervention (i.e., Fall benchmark) characteristics/ demographics for students in classrooms participating in the 
study. A baseline analysis, which confirms the integrity of the matching procedures (i.e., establishes the equality of the program 
and comparison students at the start of the first semester in Fall 2013), is also provided. 

Finally, data collection methods and the use of administrative data are discussed, along with the analytic methods and statistical 
models for the quasi-experimental design study. The chapter concludes with a discussion of the limitations of the Minnesota 
Reading Corps PreK outcome evaluation. 

A. Evaluation Logic Model 

A logic model for the Minnesota Reading Corps program illustrating key program and school inputs and activities, as well as the 
program’s desired short-term outcomes and long-term goals is provided in Appendix A.1. 34 The Minnesota Reading Corps logic 
model was developed jointly by the evaluation team and Minnesota Reading Corps program staff and served as the conceptual 
framework for the design of the PreK outcome evaluation. The logic model presents a comprehensive illustration of the complete 
Minnesota Reading Corps program, and includes inputs, activities, short-term outcomes and long-term goals for four primary 
program constituencies: PreK students, K-3 students, AmeriCorps members, and schools. 

The focus of the PreK outcome evaluation was to assess the effect of Minnesota Reading Corps program participation on 
preschool students’ emergent literacy skills. As such, the evaluation focused on only those components of the logic model 
relevant to PreK students. These components include Minnesota Reading Corps program and school-based inputs and 


33 Bollman, K. & Silberglitt, B. (2009). Minnesota Reading Corps Final Evaluation 2008-2009. Minneapolis, MN: Minnesota Reading Corps.; 
Bollman, K. & Silberglitt, B. (2011). Minnesota Reading Corps Final Evaluation 2010-2011. Minneapolis, MN: Minnesota Reading Corps. 

34 A comprehensive description of the Minnesota Reading Corps logic model is available in the following two reports: Feasibility Study of the 
Minnesota Reading Corps (2013) and Process Assessment of the Minnesota Reading Corps (2013) 
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resources, Internal Coach and AmeriCorps member coaching and supervision, AmeriCorps member individualized literacy 
interventions for students, and student proficiency outcomes. 

Four key Minnesota Reading Corps program and school-based inputs and resources are essential to successful PreK program 
implementation: 1) Minnesota Reading Corps program selection of schools based on degree of student need and school 
capacity to partner effectively with the program; 2) school identification of Tier 2 and 3 PreK students within the classroom based 
on benchmark assessment of students' emergent literacy skills; 3) web-based data management systems to track and monitor 
student progress with literacy interventions (i.e., edSpring); and 4) school use of research-based core literacy curriculum. 

In addition to the Minnesota Reading Corps program and school-based inputs, three important Minnesota Reading Corps 
program inputs related to AmeriCorps members included: 1) joint Minnesota Reading Corps and school recruitment, screening 
and placement of members in schools; 2) comprehensive Minnesota Reading Corps training of members and Internal Coaches 
in literacy interventions, assessment, data-driven decision-making and program rules; and 3) school identification and 
assignment of dedicated Internal Coaches to support and monitor the members. The logic model also illustrates the multiple 
layers of supervision and coaching the Minnesota Reading Corps program provides to its school-based Internal Coaches and 
AmeriCorps members. 

As shown in the logic model, the Minnesota Reading Corps PreK program's primary activities include: 1) supporting classroom 
instruction and enriching the literacy environment within the classroom; 2) conducting benchmark testing three times per year 
(Fall, Winter and Spring) to identify students in need of small group or one-to-one intervention (i.e., Tier 2 and 3 students); 3) 
delivering interventions to eligible students; 4) assessing and charting monthly student progress on age-specific emergent 
literacy skills using the IGDI assessment and edSpring (i.e., progress monitoring); 5) “exiting” students from Tier 2 or 3 
interventions once they achieve assessment score targets; and 6) identifying any new students eligible for small group 
interventions. The intended short-term outcomes of these activities are demonstrated improvement on IGDI measures at the 
subsequent benchmarking period (i.e., Winter, Spring). The desired long-term outcome of the Minnesota Reading Corps PreK 
program is for preschool students to achieve Kindergarten-ready targets by the end of the school year. 

Thus, when reviewing the logic model, note that the PreK outcome evaluation assessed the cumulative effect of three specific 
program elements on PreK students’ emergent literacy outcomes: 1) the Minnesota Reading Corps central operations and 
school-based inputs and resources; 2) Internal Coach and AmeriCorps member coaching and supervision; and 3) AmeriCorps 
member whole class enrichment and Tier 2 and 3 interventions. 

B. PreK Outcome Evaluation Research Questions 

As the logic model presented above illustrates, the Minnesota Reading Corps program's short-term objective is to improve 
students' literacy skills so they are Kindergarten-ready. The primary goal of the Minnesota Reading Corps PreK outcome 
evaluation was to independently assess the effect of the program on PreK students' emergent literacy scores. To achieve this 
goal, the PreK outcome evaluation focused on the following three research questions: 

RQ1 : What is the effect of the Minnesota Reading Corps program on student literacy outcomes? 
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RQ2: Does the effect on literacy outcomes vary by student characteristics/demographics? 

RQ3: Do effects on literacy outcomes vary by site/school? 35 

To explore these research questions, the evaluation team analyzed age-appropriate literacy assessment scores from a sample 
of 1,534 PreK students enrolled at 50 diverse public elementary schools, private community-based institutions, and Head Start 
programs during the 2013-2014 school year. Half of the sites (25) were experienced implemented 36 of the Minnesota Reading 
Corps program (i.e., program sites). The remaining 25 sites (i.e., comparison sites) were selected for participation in the study 
based on a prescribed set of educationally significant baseline characteristics which matched those of the sampled Minnesota 
Reading Corps program sites (e.g., urbanicity, program type, student age mix, etc.). Thus, 25 pairs of sites (i.e., 25 program sites 
and 25 comparison sites) participated in the evaluation. Students who attended Minnesota Reading Corps program sites were 
enrolled in PreK classrooms where one or more AmeriCorps members served. Data from the Fall, Winter, and Spring 
benchmarks were used to answer the evaluation's primary research questions, which estimate the effect of the Minnesota 
Reading Corps program on students' emergent literacy outcomes. 

C. Site Selection 

The site selection process was designed to ensure representation of the diverse schools participating in the Minnesota Reading 
Corps PreK program. Therefore, site selection was both purposive and involved a multi-step stratified sampling process. Since 
the Minnesota Reading Corps program began 10 years ago in 2003 and expanded rapidly over the years, there was a wide 
range in the number of years sites had participated in the program. As it was not our intention to assess how well sites 
adopted/executed the program during their first year of participation, the site sampling process began by limiting the sampling 
frame to only sites that had fully implemented the Minnesota Reading Corps PreK program for at least one year. Next, to ensure 
geographical diversity, sites were stratified by urbanicity (i.e., urban, suburban, and rural), institution type (public school, 
community-based program and Head Start), 37 and student ages served within a classroom (4- and 5-year old students only or a 
mix of 3-, 4-, and 5-year old students). 38 Thus, the sampling frame consisted of 175 sites, with 12 stratum for sampling within 
urbanicity, institution type, and age group. 39 A total of 40 sites were selected, using Probability Proportional to Size (PPS), 


35 We were also interested in examining the effects of AmeriCorps member type (i.e., Educator Corps or Community Corps) on student 
emergent literacy outcomes. However, given the small sample size among Minnesota Reading Corps PreK sites (N=25) participating in this 
study, as well as the fact that we did not specifically sample for member type, we could not examine this question. We did conduct a balance 
test among the 25 program schools to determine if it might be possible to conduct an exploratory analysis. Unfortunately we found significant 
lack of balance in the types of members at sites, preventing us from conducting such an analysis. 

36 That is, the program had been in operation at the site for at least one full school year. 

37 Although there were originally five types of institutions, including private and charter schools, only public schools, community-based, and 
Head Start programs were included in the sample. Only four schools were charter and private schools and could not be easily combined with 
other categories due to the fact that they are not similar to any of the other school types. Therefore, these four schools were deleted from the 
sampling frame. 

38 Only three schools served 3-year old students only, so they were excluded from the sampling frame. 

39 When a crosstab of schools was run by institution type and urbanicity, there were very few schools in some categories. As a result, suburban 
community-based, suburban head start, and rural community-based schools were deleted from the sampling frame; they only comprised seven 
schools in total. 
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whereby larger sites based on student enrollment (defined as the number of PreK students served by the Minnesota Reading 
Corps during the 2012-2013 school year) had a higher probability of selection. Using PPS ensured a statistically adequate 
sample size to conduct the PreK outcome evaluation. It is important to note that while the Minnesota Reading Corps program 
staff encouraged sites to participate, participation in the evaluation was voluntary. In the end, 39 sites agreed to participate in the 
PreK outcome evaluation during the 2013-2014 school year. 40 

To identify a comparison group for the PreK QED, the study followed U.S. Department of Education's Institute of Education 
Sciences' (IES) What Works Clearing House (WWC) guidelines for a QED that “Meets Evidence Standards with Reservations.” 
PreK programs were eligible for consideration for the comparison group if: 1) the program had no history of implementing the 
Minnesota Reading Corps program; and 2) the program staff had not been trained in the SEEDS of Emergent Literacy 
approach. 41 From this group of eligible PreK sites, a comparison group was identified by matching the site to a Minnesota 
Reading Corps PreK program site on a select group of educationally important pre-intervention characteristics 42 (i.e., urbanicity, 
institution type, classroom composition of student ages, percentage of students eligible for FRPL, 43 and percentage of students 
who were DLLs). Additional secondary variables were also considered when matching program and comparisons sites (e.g., site 
enrollment, instructional time, and student to teacher ratio), though they were primarily used to sharpen the match when multiple 
potential match sites were identified. The final set of variables is provided in Exhibit 111.1. 


Table 111 . 1 . PreK QED Matching Variables 


Order 

Variable 

Description 

Primary Matching Variables 

1 

Urbanicity 

Urban, suburban or rural 

2 

Institution Type 

Public school, Head Start program, Community-based program 

3 

Classroom Composition of 
Student Ages Served 

Only 4/5-year old students or mixed 3-, 4- and 5-year old students 

4 

Socio-Economic Status 

% students eligible for Free or Reduce Price Lunch (FRPL) and/or % poverty of families 
with children in the census tract within which the site is located 

5 

Dual Language Learners (DLL) 
Status 

% of students who do not speak English as a first language. 

Secondary Matching Variables 

6 

Site Enrollment 

Number of students enrolled in the PreK program at the site. 

7 

Instructional Time 

Number of hours of instruction per week (e.g., half vs. full day; number of days per week) 

8 

Student to Teacher/Adult Ratio 

The number of students for every teacher or adult in the classroom. 


40 The school that did not participate referenced staff shortages as the reason. 

41 Since SEEDS is an integral part of the Minnesota Reading Corps program, including SEEDS trained sites in the evaluation would have 
eliminated the possibility of detecting an effect of this critical program element. 

42 These variables were based on WWC recommendations, administrative data collected by Minnesota schools at the time, the literature on 
predictors of preschool emergent-literacy outcomes, and consultation with the project's Technical Working Group (TWG). 

43 A measure of household poverty and socio-economic status 
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Once potential comparison sites were identified, the evaluation team approached site administrators to gain their participation in 
the study through a prescribed recruitment process. Several sites agreed to participate because they would benefit from 
obtaining access to the assessment data we collected on their students after the study period was complete. In the end, the 
study team was able to match 29 schools to Minnesota Reading Corps programs and recruit 25 sites to participate in the study. 
Thus, 25 pairs (25 program sites and 25 comparison sites) for a total of 50 sites participated in the outcome evaluation. The total 
student sample size was determined by the number of students enrolled in a participating PreK classroom at one of the 50 
selected sites. If a comparison site had more classrooms and students than was required to achieve sufficient statistical power 
(i.e., 30 students across two classrooms), the comparison site classrooms that best mirrored the matched-Minnesota Reading 
Corps program classroom on amount of instructional time (i.e., days per week, hours per day) were selected. The list of 
participating schools and their key characteristics are provided in Table 111.2 below. 
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Table 111.2. Characteristics of schools participating in the Minnesota Reading Corps PreK Outcome Evaluation (Fall 2013) 


School 

MRC 

Location in 
Minnesota 

Urbanicity 1 

Program 

Type 

Classroom Type 

% DLL 

% FRPL 

School 

Enrollment 

2012-13 

Study 

Participants 

2013-14 

(N) 

Child Care Resource and Referral Head Start: 
Brookside 

D 

Southeast 

Rural 

Head Start 

Mix of 3-, 4-, and 5-year old 
students 

46 

100 

75 

65 

Austin Head Start 


Southeast 

Rural 

Head Start 

Mix of 3-, 4-, and 5-year old 
students 

55.5 

100 

68 

24 

Kootasca Head Start: Itasca Resource Center 

D 

Northcentral 

Rural 

Head Start 

4- and 5-year old students 
only 

0 

100 

34 

31 

Backus Head Start 


Northcentral 

Rural 

Head Start 

3- year old students; 4- and 
5-year old students only 

0 

100 

42 

26 

Duluth Head Start: Stowe Elementary 

D 

Northeast 

Rural 

Head Start 

Mix of 3-, 4-, and 5-year old 
students 

0 

100 

32 

30 

Bemidji Head Start 


Northcentral 

Rural 

Head Start 

Mix of 3-, 4-, and 5-year old 
students 

0 

100 

98 

28 

Kootasca Head Start: Deer River (King Elementary) 

D 

Northcentral 

Rural 

Head Start 

4- and 5-year old students 
only 

0 

100 

34 

17 

Blackduck Head Start 


Northcentral 

Rural 

Head Start 

4- and 5-year old students 
only 

13 

100 

30 

12 

Minneapolis Public Schools: Waite Park Community 

D 

Metro 

Urban 

Public 

4- and 5-year old students 
only 

18 

70 

40 

34 

Brimhail & Emmett Eiementaries 


Metro 

Urban 

Public 

4- and 5-year old students 
only 

32 

75 

30 

29 

Becker Early Childhood 

B 

Central 

Rural 

Public 

Mix of 3-, 4-, and 5-year old 
students 

1 

19 

84 

80 

Cannon Fails Preschool 


Southeast 

Rural 

Public 

Mix of 3-, 4-, and 5-year old 
students 

6.7 

18 

95 

18 

Albany Area Schools: Early Childhood Center 
(Avon) 

0 

Central 

Rural 

Public 

4- and 5-year old students 
only 

0 

30 

49 

52 

Cold Spring Elementary 


Central 

Rural 

Public 

4- and 5-year old students 
only 

17 

31 

84 

23 

Saint Paul Public Schools: Heights Community 
School 

D 

Metro 

Rural 

Public 

4- and 5-year old students 
only 

55 

- 

40 

39 
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School 

MRC 

Location in 
Minnesota 

Urbanicity 

Community of Peace 


Metro 

Urban 

Elk River Public Schools: Rogers Elementary 

D 

Metro 

Suburban 

Delano Wee Tiger 


Metro 

Suburban 

Probsfield Early Learning 

0 

Northwest 

Rural 

Dilworth Elementary 


Northwest 

Rural 

Community Action Partnerships of Ramsey & 
Washington Counties: Mt. Airy 

1] 

Metro 

Urban 

Families Together 


Metro 

Urban 

Duluth Head Start: Laura MacArthur 

D 

Northeast 

Rural 

Hermantown Head Start 


Northeast 

Rural 

YWCA of Minneapolis: Downtown 

0 

Metro 

Urban 

Joy Academy 


Metro 

Urban 

New London Spicer: Cub Kids 

D 

Southwest 

Rural 

Humphrey Elementary 


Metro 

Rural 

Burnsville Public School District: Diamondhead 
Education Center 

0 

Metro 

Suburban 

Longfellow Elementary 


Southeast 

Suburban 

North Branch Early Childhood: North Branch Early 
Childhood 

D 

Central 

Rural 
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Program 

Type Classroom Type 


Public 

4- and 5-year old students 
only 

Public 

4- and 5-year old students 
only 

Public 

4- and 5-year old students 
only 

Public 

4- and 5-year old students 
only 

Public 

4- and 5-year old students 
only 

Head Start 

Mix of 3-, 4-, and 5-year old 
students 

Head Start 

Mix of 3-, 4-, and 5-year old 
students 

Head Start 

Mix of 3-, 4-, and 5-year old 
students 

Head Start 

Mix of 3-, 4-, and 5-year old 
students 

Community 

Mix of 3-, 4-, and 5-year old 
students 

Community 

Mix of 3-, 4-, and 5-year old 
students 

Public 

Mix of 3-, 4-, and 5-year old 
students 

Public 

Mix of 3-, 4-, and 5-year old 
students 

Public 

Mix of 3-, 4-, and 5-year old 
students 

Public 

Mix of 3-, 4-, and 5-year old 
students 

Public 

Mix of 3-, 4-, and 5-year old 
students 


% FRPL 

School 

Enrollment 

2012-13 

Study 

Participants 

2013-14 

(N) 

31 

42 

26 

23 

53 

31 

14 

114 

28 

11 

75 

33 

30 

40 

36 

40 

- 

74 

100 

44 

26 

100 

64 

63 

8 

33 

36 

45 

32 

43 

51 

20 

9 

26.4 

20 

44 

28 

68 

37 

38.2 

121 

164 

48 

- 

29 

27 

167 

141 


26 

6 

0 

1 

0 

78 

0 

0 

5.3 

3 

0 

1 

56 

23 

5 
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School 

MRC 

Location in 
Minnesota 

Urbanicity 

Fergus Fails Public Schools: Otter Preschool 


Northwest 

Rural 

Oakdale Elementary 

D 

Metro 

Suburban 

Melrose Elementary 


Central 

Suburban 

Kerkhoven-Murdock Sunburg School District: 
Murdock 

D 

Southwest 

Rural 

Paynesville Elementary 


Southwest 

Rural 

Pine City Elementary 

D 

Northeast 

Rural 

Pillager Elementary 


Central 

Rural 

TCU (TriCity United) School Readiness: Lonsdale 
School Readiness 

D 

Metro 

Rural 

Rainbow Preschool 


Metro 

Rural 

YMCA of Greater St. Paul: Midway 

0 

Metro 

Urban 

St. Paul Eastside YMCA 


Metro 

Urban 

Fair Oaks Elementary 

D 

Metro 

Suburban 

Talahi Elementary 


Central 

Suburban 

Duluth Head Start: Meyers Wilkens Elementary 

D 

Northeast 

Rural 

Walker Head Start 


Northcentral 

Rural 

Elk River Public Schools: Zimmerman Elementary 

D 

Metro 

Suburban 


OUTCOME EVALUATION OF THE MINNESOTA READING CORPS PREK PROGRAM 


The Corporation for National and Community Service 1 2015 


Program 

Type Classroom Type 


Public 

Mix of 3-, 4-, and 5-year old 
students 

Public 

4- and 5-year old students 
only 

Public 

4- and 5-year old students 
only 

Public 

4- and 5-year old students 
only 

Public 

4- and 5-year old students 
only 

Public 

4- and 5-year old students 
only 

Public 

4- and 5-year old students 
only 

Public 

Mix of 3-, 4-, and 5-year old 
students 

Public 

Mix of 3-, 4-, and 5-year old 
students 

Community 

4- and 5-year old students 
only 

Community 

3- year old students; 4- year 
old students; Mix of 3-,4- 
year old students 

Public 

4- and 5-year old students 
only 

Public 

4- and 5-year old students 
only 

Head Start 

Mix of 3-, 4-, and 5-year old 
students 

Head Start 

Mix of 3-, 4-, and 5-year old 
students 

Public 

4- and 5-year old students 
only 


% FRPL 

School 

Enrollment 

2012-13 

Study 

Participants 

2013-14 

(N) 

43 

200 

30 

32 

32 

34 

50 

29 

34 

27 

31 

42 

38 

36 

26 

43 

27 

29 

54 

29 

28 

43 

49 

47 

25 

55 

34 

85 

20 

19 

- 

78 

17 

83 

36 

18 

91.7 

- 

28 

100 

62 

65 

100 

30 

24 

23 

68 

31 


3 

32 

33.3 

2 

0 

0 

0 

0 

1 

10 

.05% 

50 

39.4 

0 

0 

8 
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School 

MRC 

Location in 
Minnesota 

Urbanicity 1 

Winsted Elementary 


Metro 

Suburban 

Saint Paul Public Schools: Dayton's Bluff 
Elementary 

D 

Metro 

Urban 

Zanewood Elementary 


Metro 

Urban 
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Program 

Type 

Classroom Type 

% DLL 

% FRPL 

School 

Enrollment 

2012-13 

Study 

Participants 

2013-14 

(N) 

Public 

4- and 5-year old students 
only 

4.4 

42 

- 

30 

Public 

4- and 5-year old students 
only 

48 

91 

40 

35 

Public 

4- and 5-year old students 
only 

41.3 

83 

40 

23 
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Table 111.3 presents descriptive statistics by age for the PreK students included in the evaluation. Demographics include gender, 
race/ethnicity, Dual Language Learner (DLL) status, and Fall benchmark scores. During the school year, some students left the 
school area (i.e., moved) or were chronically absent and did not complete all three benchmark assessments. These students 
were removed from the analytic sample. Thus, the final sample of students included in the evaluation totaled 1 ,534 students. 44 

The evaluation team conducted a power analysis prior to developing the evaluation's sampling plan, in which the number of 
students required to detect a difference between program and comparison groups of students was calculated. The power 
analysis relied on the student, member, and site population numbers reported by the Minnesota Reading Corps program from 
the 2012-2013 school year. For this analysis, we estimated the minimum detectable effect (MDE) for a given number of schools, 
assuming 30 students per program and comparison school would participate in the evaluation. The power analysis 
employed previous Minnesota Reading Corps program data on PreK student-level outcomes to estimate the necessary design 
parameters (specifically, the effects of clustering students in schools). Our analysis revealed that, assuming power of 0.80 and 
an alpha level of 0.05, 25 schools in each condition (program and comparison) would produce an MDE of about 0.2 standard 
deviations. Thus, assuming 25 schools with 30 students in each school would result in a required sample size of 750 students to 
detect a substantively significant difference between program and comparison groups. As with most cluster level effects, the 
power results are relatively robust to minimal loss of students between data collection periods; however, the MDE increases as 
the number of schools decrease (that is, we lose sensitivity as we lose schools participating in the study). 


44 For each grade, we demonstrate low levels of attrition assuming the “liberal” standard outlined in the WVVC Evidence Review Protocol for 
Early Childhood interventions (Version 2): http://ies.ed.gov/ncee/wwc/pdf/reference resources/ece protocol v2.0.pdf 
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Table III. 3. Program and comparison student participants for the Minnesota Reading Corps PreK Outcome Evaluation 
(Fall 2013) 



3-year old students 
(N= 286) 

4- and 5-year old students 
(N= 1,480) 


Percent 

Percent 

Female 

48% 

47% 

Race/Ethnicity 



White 

52% 

66% 

Black 

15% 

11% 

Asian 

10% 

7% 

Hispanic 

10% 

8% 

Other 

14% 

9% 

Dual Language Learner (DLL) 

18% 

14% 


Table 111.3 shows that demographics somewhat varied by age group. Among 3-year old students, a little more than half were 
White (52%), 15% Black, and 14% “Other.” Only 10% of 3-year old students were Asian and 10% Hispanic. Among 4- and 5-year 
old students, an even higher percentage of students were White (66%) with smaller percentages of children of other races 
represented (11% Black, 7% Asian, 8% Hispanic, and 9% “Other”). These statistics are not unexpected given that the state of 
Minnesota has higher percentages of White students than the broader American population. 45 Also, a relatively small percentage 
of the analytic sample included students who were classified by their sites as Dual Language Learners (DLLs), which was not 
unexpected given the low percentage of minority children in the sample. 

Table 111.4 compares key demographic variables between students enrolled in the Minnesota Reading Corps program and 
comparison sites, including average Fall benchmark (baseline) scores. The 4- and 5-year old students at program and 
comparison sites showed no significant differences on key demographic variables, thus confirming that sites were adequately 
matched on most variables. For 3-year old students, some differences were found, specifically regarding smaller racial groups 
and DLL status. However, these differences may result from the substantially smaller sample sizes for 3-year old students. While 
few major differences were found by student demographics, some comparison sites reported higher average Fall benchmark 
scores than their matched Minnesota Reading Corps sites on some outcome measures. As a result, our statistical approach 
controlled for the Fall benchmark scores to account for this imbalance. Importantly, the comparison group students posted 
slightly higher Fall benchmark scores on average than students in the program group, so these differences at baseline would 
benefit the comparison sites rather than Minnesota Reading Corps program sites. 


45 http://auickfacts.census.aov/gfd/states/27000.html 
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Table 111.4. Differences between Comparison and Program group students by age (Fall 2013) 



Key: C=Comparison, P=Program (Minnesota Reading Corps), SE=Standard Error 
*p<05; ”p<01 

D. Data Collection 

As previously discussed, IGDI assessment scores are the primary sources of data for PreK student emergent literacy outcomes 
at Minnesota Reading Corps sites. The evaluation team obtained all benchmark assessment data from the Minnesota Reading 
Corps program for all 25 Minnesota Reading Corps PreK sites selected for participation in the evaluation. As mentioned 
previously, AmeriCorps members collect benchmark data three times a year on all students in their Minnesota Reading Corps 
PreK classrooms to determine eligibility for Tier 2 or 3 small group or one-to-one intervention. Given that Members already 
collect this data, and so as not to duplicate assessment efforts, the evaluation team requested access to the 2013-2014 Fall, 
Winter, and Spring benchmark data for our analysis of student outcomes. 

The evaluation team obtained benchmark assessment data directly from the Minnesota Reading Corps program. At the end of 
the 2013-2014 school year, the complete dataset was de-identified and exported to the evaluation team for analysis. The 
Minnesota Reading Corps provided the evaluation team with other administrative data including student and AmeriCorps 
member demographics. All assessment and demographic data was combined into a single dataset that is the source of all 
analyses and results presented in this report. 

The evaluation team was responsible for collecting equivalent IGDI data three times a year (Fall, Winter, Spring) from PreK 
students at the comparison sites. In order to collect this data, the evaluation team sent field testers, who would be responsible for 
assessing students at comparison sites, to the same Minnesota Reading Corps IGDI data collection training as that attended by 
AmeriCorps members. This process ensured that the field testers: 1) used the same IGDI data collection instruments for 
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collecting student level benchmarks on comparison site students as those used to collect data from students at the Minnesota 
Reading Corps program sites; and 2) received the same standardized training in assessment administration and scoring as the 
AmeriCorps members. This common use of instrumentation and training ensured commonality of assessment practices across 
both program and comparison sites. 

In addition to the use of common measures and receipt of common training, whenever possible the same field testers and 
research staff were assigned to collect data from the same students at the same sites throughout the school year. This allowed 
the field testers and research staff to build strong and trusting working relationships with the administrators, teachers and 
students at participating comparison sites. The students at comparison sites became familiar and comfortable with the field 
testers over the course of the year, which further facilitated testing and data collection. 

Upon completion of data collection, the student-level IGDI data collected from comparison sites were combined for analysis with 
de-identified IGDI data from the Minnesota Reading Corps dataset described earlier. The resulting dataset contained all the 
assessment and demographic data collected from comparison sites and existing data extracted for participating Minnesota 
Reading Corps program sites. The final dataset was processed, cleaned, and coded in preparation for data analysis. 

E. Analysis 

In this section, we provide a summary of the methods and analytic approaches used to analyze the student assessment data 
collected in Fall, Winter, and Spring. As discussed above, each Minnesota Reading Corps program site was paired with a 
comparison site on several key variables. During data analysis, these variables were equated using statistical controls so as to 
adjust for pre-intervention differences among groups. This produced a “site pair,” comprised of one Minnesota Reading Corps 
program site and one matched comparison site. The hierarchy of the data is illustrated in Figure I II. 5, where each site pair 
contains two sites, and each site contains a set of students. The analysis was conducted separately for 3-year old students and 
then for 4- and 5-year old students, because 1) 3-year old students are developmental^ distinct from 4- and 5-year old students, 
2) there is more variability in prior exposure to PreK among 4- and 5- year old students than among 3-year old students, and 3) 
the letter name assessments differed for 3-year old students versus 4- and 5-year old students. A more detailed methodology 
section is provided in Appendix A.3. As explained in our presentation of the research questions in section III.B above, the 
analysis was structured to answer three primary research questions, which each address the effect of the program on PreK 
students’ emergent literacy outcomes. Other analyses included robustness checks on the variability of the treatment effects and 
an intraclass correlation analysis of Fall to Spring gains. 
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Figure 111.5. Illustration of PreK Evaluation Data Hierarchy 



Outcome Analysis 

The overall goal of the study was to obtain a measure of the average treatment effect of the Minnesota Reading Corps program 
within each site pair by each student age group. We examined differences by employing an ANCOVA (Analysis of Covariance) 
regression using fixed effects for site pairs. The ANCOVA regression predicts the Winter or Spring benchmark score (outcome) 
as a function of the student level covariates, namely the Fall benchmark score and age (3-year old and 4/5-year old), in addition 
to the site level Minnesota Reading Corps program participation variable (indicator). The Minnesota Reading Corps indicator 
coefficient is then the average difference in the difference between the dependent score (Winter or Spring) and the Fall score for 
an average student. By centering the Fall score on the grand mean, the intercept of the model is the growth for students at 
comparison sites, and the coefficient for the Minnesota Reading Corps indicator is the difference in the growth for program 
students relative to comparison students. 

Two types of subgroup analyses were employed by the evaluation team. First, we added specific covariates and moderators to 
the ANCOVA model and jointly tested the effects to determine if there were significant differences in effects across major 
subgroups, implying that the Minnesota Reading Corps program may be more or less beneficial for selected subgroups of 
students. For the second analysis (for 4- and 5-year old students only), we used the moderators of the major subgroup indicators 
(e.g., females, males, etc.) to estimate the net average treatment effect of the Minnesota Reading Corps program for 4- and 5- 
year old students, essentially subdividing the data by major subgroup and applying the same analysis as for the entire sample. 
Due to sample size constraints we were unable to conduct subgroup analysis on the 3-year old student sample. Subgroups were 
defined based on characteristics provided at baseline and each level was determined by individual characteristics, including 
gender, race/ethnicity, and Dual Language Learner (DLL) status. 

Robustness Analysis 

The evaluation team also tested the variability of the program effect at the assigned site pair level. For this analysis, we 
estimated the Minnesota Reading Corps effect between program and comparison sites, and then estimated whether, on 
average, this difference was positive. The findings from the robustness analysis are provided in Appendix C. 
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Site-Level Effects on Assessment Scores 

The Minnesota Reading Corps PreK program operates in a diverse range of educational contexts (i.e., public schools, Head 
Start centers, and community-based preschools). An important research question for the Minnesota Reading Corps program is 
whether there are differences in student-level effects of the program depending by site type. Therefore, we conducted subgroup 
analysis on one site-level variable-- whether students attended a public school or a Head Start program. We did not have enough 
community-based sites in the sample to conduct subgroup analyses on students attending this type of site. Given the diversity of 
early childhood education programs (e.g., curricula, attendance time, student/family demographics, funding source, etc.), it was 
unclear if and how the Minnesota Reading Corps PreK program might interact with the type of site (i.e., public school or Head 
Start center). The results of this analysis allowed us to examine whether the Minnesota Reading Corps PreK program is more or 
less effective in public preschools or Head Start centers. 

One additional analysis was performed to examine whether student outcomes varied by site-- an intraclass correlation (ICC) 
analysis was conducted to determine whether pair-level random effects influenced program impacts for students attending a 
Minnesota Reading Corps PreK program site. In other words, we examined whether there was variation in the effect of the 
program on students across the 25 pairs of sites (program vs. comparison). The site pair-level ICC measures the degree to 
which students, who attended one site within the same pair, correlate with students in other site pairs. We examined pair-level 
effects rather than site-level effects because Minnesota Reading Corps program and comparison sites were matched on key site 
characteristics prior to baseline. 

F. Limitations of the Study 

The primary objective of the Minnesota Reading Corps PreK evaluation was to assess the effect of the Minnesota Reading 
Corps program on preschool students’ emergent literacy outcomes. In order to achieve this objective, the evaluation was 
designed to measure Minnesota Reading Corps program effects in a rigorous quasi-experimental design study. However, 
unavoidable limitations inherent in the study design and in working with schools and students constrained some aspects of the 
evaluation's design, implementation, and analysis. 

The single most important limitation of the study design is its inability to control for unobserved differences between the 
Minnesota Reading Corps program and comparison sites. Despite implementing a thoughtful approach to the matching process 
using educationally significant baseline measures, it is not possible to obtain or match on all possible site-level factors which may 
bias student outcomes. For example, the strongest predictor of student performance on outcome measures is their performance 
at baseline (i.e., Fall benchmark [pretest] score). Our matching process was conducted without access to any baseline data 
because it had yet to be collected, thereby, possibly resulting in bias within our site-level pairs. To examine these differences and 
test the integrity of our matching process, we conducted statistical tests of average Fall scores within pairs (simple t-tests) and 
found the vast majority of our site-level pairs did not result in significant differences at baseline across the five outcome 
measures. For further information on our balance tests (school matching validity analysis), please see Appendix A.2. 

Another major constraint is that site participation in the evaluation was voluntary. As such, it is plausible that our estimates may 
not generalize to the population. We first addressed this issue by using stratified random sampling to select Minnesota Reading 
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Corps PreK sites; however, we were unable to find reasonable matches for all of our sampled schools resulting in a possibly 
biased sample. We conducted sensitivity analyses to better identify differences between the Minnesota Reading Corps sites that 
participated in the study and those that did not, mainly due to our inability to identify a matched comparison school. The findings 
from these analyses, available in Appendix A.2, confirm the representativeness of our sample. 

Unequal sample sizes across the two age groups (3-year old vs. 4- and 5-year old) were another program-induced constraint 
that resulted in differences in statistical power to detect program effects within age groups. The different sample sizes by age 
were unavoidable as very few sites in our original sampling frame had 3-year old only classrooms. This forced us to stratify the 
sample by the two larger groups of 4- and 5-year old classrooms and mixed 3-, 4-, and 5-year old classrooms; thus, less 3-year 
old students participated in the study. 

Another limitation inherent in the execution of the evaluation was the necessity to assess students three times during the school 
year at both the program and comparison sites. It is unclear what effect if any the act of being assessed might have had on 
student outcome scores. It is possible that simply participating in these assessments might positively affect student proficiency. If 
such a test effect exists it is likely to be small. Further, the effect should benefit the comparison group, increasing their scores 
above what one would typically expect had they not been assessed. 

More common limitations in education studies that the evaluation team anticipated included: student attrition, data integrity, and 
tracking alternate interventions. As mentioned above, the evaluation team implemented a quasi-experimental design using 
matched comparison sites. Benchmark assessment scores were then taken three times during the school year on all students 
within participating classrooms and sites. Statistical tests showed that students in the program and comparison groups did not 
differ on important demographic variables and proficiency measures at the beginning of the school year. As is inevitable, 
students moved away or were chronically absent (i.e., attrition). When this occurred, the affected student was dropped from the 
study's analytic sample. The deletion of cases was moderate, reducing the sample size from 1 ,766 to 1 ,534 students (a loss of 
about 24 percent) and could result in an imbalance between the program and comparison groups. However, a more central 
concern is the differential attrition between program and comparison groups. The comparison sites had attrition of about 18 
percent of students whereas the Minnesota Reading Corps program sites had about 27 percent attrition. For a more thorough 
analysis of the differences between program and comparison groups, please see our analysis developed for the What Works 
Clearinghouse (WWC) located in Appendix D. 

Another common limitation related to interpretation of outcome evaluation results is concerns as to whether alternative 
interventions similar to the treatment (in this case, the Minnesota Reading Corps' Literacy Rich Schedule and SEEDS approach 
to instruction) could similarly affect student outcomes at comparison programs. To overcome this limitation, the evaluation team 
excluded sites where staff had been trained in the SEEDS of Emergent Literacy, which is an integral part of the Minnesota 
Reading Corps PreK program. No other similar alternative interventions were identified at the PreK comparison sites. 
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Study Findings 

This section of the report presents the findings from the analysis of IGDI assessment data collected in Fall and Spring from PreK 
students enrolled at the program or comparison sites during the 2013-2014 school year. The findings from the analysis of Fall 
and Winter benchmark scores are provided in Appendix B.l. As discussed earlier, the program group consisted of preschool 
children who attended a classroom where the Minnesota Reading Corps PreK program was being implemented during the 2013- 
2014 school year, while the comparison group consisted of preschool students who attended sites that had not participated 
previously or currently in the program. 

To address the first research question (RQ1) of whether the Minnesota Reading Corps program had an effect on PreK students' 
literacy outcomes, the evaluation team compared the Spring assessment scores of students who attended a program site to the 
scores of those who attended a similar comparison preschool site that was not implementing the Minnesota Reading Corps. As 
discussed in Chapter III, the evaluation team estimated the average treatment effect by developing a single ANCOVA model for 
each outcome measure and age group (3-year old students and 4- and 5-year old students). In addition to providing estimated 
differences between program and comparison groups, we also present standard errors, confidence intervals and statistical tests. 

Below the evaluation team provides the findings from the Spring benchmark scores across the five IGDI outcome measures by 
age group. (Findings for Winter benchmark scores are provided in Appendix B.l.) Following the presentation of findings for all 
students, we provide the results of our subgroup analysis for gender, race/ethnicity, and DLL status (RQ2). We conclude this 
chapter with the presentation of results from our examination of student effects by AmeriCorps member type and site (RQ3). 

More detailed tables on the models used are provided in Appendix B.2. 

Overall Findings for 4- and 5-year Old Students 

The Minnesota Reading Corps PreK program had statistically significant and large effects across all five IGDI outcome measures 
of emergent literacy skills for 4- and 5-year old students. Below we provide a discussion of the findings in growth between Fall 
and Spring benchmarks for 4- and 5-year old students. To better illustrate the findings on each outcome measure, we also 
provide several figures showing the average scores on the Fall, Winter, and Spring benchmarks for PreK students in Minnesota 
Reading Corps program and comparison site classrooms. Average growth for the school year, which is provided in Table IV.1, is 
calculated by taking the difference between the Spring and Fall scores for program and comparison site students, while 
controlling for the Fall (baseline) score. We refer to the findings in both Table IV.1 and the various figures in reporting the study 
findings. 


Table IV.1. Average growth among 4- and 5-year old PreK students by Spring 


IGDI Fluency Assessment 

Treatment 

Assignment 

N 

Average 

Growth 

SE 

Effect Size 

Confidence Interval 


Program+ 

783 

8.22*** 

0.33 




Letter Sounds 

Comparison 

484 

4.42*** 

0.58 





Difference 

1267 

3.80*** 

0.90 

0.71 

0.36 

1.05 
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Rhyming 

Program 

Comparison 

Difference 

789 

553 

1342 

7.28*** 

3.77*** 

3.51*** 

0.23 

0.31 

0.52 

0.66 

0.46 

0.87 


Program 

802 

14.01** 

0.45 




Letter Names 

Comparison 

552 

9.87*** 

0.72 





Difference 

1354 

4.15** 

1.17 

0.40 

0.17 

0.63 


Program 

793 

8.68*** 

0.28 




Picture Names 

Comparison 

550 

4.65*** 

0.39 





Difference 

1343 

4.03*** 

0.66 

0.49 

0.32 

0.65 


Program 

792 

5.37*** 

0.17 




Alliteration 

Comparison 

486 

2.68*** 

0.28 





Difference 

1278 

2.68*** 

0.44 

0.72 

0.48 

0.97 


***<.001, **<.01, *<.05; +Minnesota Reading Corps 


What is the effect of the Minnesota Reading Corps program on 4- and 5-year old students’ letter sound 
fluency? 

The Minnesota Reading Corps program had a significant and large effect on 4- and 5-year old PreK students’ letter sound 
fluency scores between Fall benchmark (September 2013) and Spring benchmark (May 2014). As shown in Table IV.1 above, 
both groups of students’ letter sound fluency scores increased significantly from Fall to Spring, however by the end of the school 
year, PreK students in a Minnesota Reading Corps classroom were able to produce an average of 3.8 more letter sounds 
correctly in a one-minute period than PreK students in a comparison classroom. This represents a large effect size of about .71 
standard deviations. Furthermore, as shown in Figure IV.1 below, by the end of the school year, the Minnesota Reading Corps 
PreK students’ average score of 10.6 letter sounds was above the Spring benchmark of 10, whereas the comparison PreK 
students’ average score of 6.8 letter sounds was well below the benchmark. 
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Figure IV.1 . Growth in Letter Sound Fluency for 4- and 5-year old PreK program and comparison students 



What is the effect of the Minnesota Reading Corps program on 4- and 5-year old students’ rhyming fluency? 

In addition to letter sounds, 4- and 5-year old PreK students in Minnesota Reading Corps classrooms had significantly higher 
scores in rhyming fluency at the end of the school year than did PreK students in comparison classrooms. Table IV.1 shows that 
both groups of students’ rhyming fluency scores increased significantly from Fall to Spring, however by the end of the school 
year PreK students in Minnesota Reading Corps classrooms were able to identify an average of 2.1 more rhymes in a two 
minute period than PreK students in comparison classrooms. This represents a large effect size of about 0.66 standard 
deviations. Figure IV.2 shows that once again the Minnesota Reading Corps PreK students' average score (13.4) reached the 
Spring benchmark of 12 words, while the comparison students' average score (8.7) did not reach benchmark by the end of the 
school year. 
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Figure IV.2. Growth in Rhyming Fluency for 4- and 5-year old PreK program and comparison students 



Time of Benchmark 


Program 

Comparison 

— —Rhyming Fluency 
Benchmark 


What is the effect of the Minnesota Reading Corps program on 4- and 5-year old students’ letter name fluency? 

As with letter sound fluency and rhyming fluency, a significant effect was also observed in letter name fluency among 4- and 5- 
year old PreK students in Minnesota Reading Corps classrooms. By the end of the school year, both groups of students’ letter 
name fluency scores increased significantly from Fall, however PreK students in Minnesota Reading Corps classrooms were 
able to identify an average of 4.2 more letters correctly in a one minute period than PreK students in comparison classrooms. 
This represents a considerable effect size of about .40 standard deviations. One noticeable difference between the findings on 
letter name fluency and other outcome measures is the performance of the 4- and 5-year old PreK students in the comparison 
group (shown in Figure IV.3). In this case, 4- and 5-year old PreK students in both Minnesota Reading Corps and comparison 
classrooms produced an average letter name fluency score (22.7 and 18.6, respectively) which well exceeded the Spring 
benchmark of 14 letters. 
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Figure IV.3. Growth in Letter Name Fluency for 4- and 5-year old PreK program and comparison students 


24 



Fall Winter Spring 

Time of Benchmark 


Program 

Comparison 

— —Letter Name Fluency 
Benchmark 


What is the effect of the Minnesota Reading Corps program on 4- and 5-year old students’ picture name 
fluency? 

The Minnesota Reading Corps program also had a significant effect on 4- and 5-year old PreK students’ picture name fluency 
scores between Fall and Spring benchmark. By the end of the school year, both groups of students’ picture name fluency scores 
increased significantly from Fall, however PreK students in Minnesota Reading Corps classrooms were able to identify an 
average of 4.0 more pictures correctly in a one minute period than PreK students in comparison classrooms. As with the other 
outcome measures, this difference represents a substantial effect size of about .49 standard deviations. Figure IV.4 below shows 
that, by the end of the school year, the Minnesota Reading Corps PreK students’ average score of 28.3 pictures was above the 
Spring benchmark of 26, whereas the comparison PreK students’ average score of 24.2 pictures was just below benchmark. 
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Figure IV.4. Growth in Picture Name Fluency for 4- and 5-year old PreK program and comparison students 



Time of Benchmark 


Program 

Comparison 

— —Picture Naming Fluency 
Benchmark 


What is the effect of the Minnesota Reading Corps program on 4- and 5-year old students’ alliteration 
fluency? 

The final outcome examined for 4- and 5- year old students was alliteration fluency, the ability to identify words beginning with 
the same letter sound (e.g., cat and car). Between Fall and Spring benchmark, both groups of students' alliteration fluency 
scores increased significantly, however PreK students in Minnesota Reading Corps classrooms were able to identify an average 
of 2.7 more alliterations correctly in a two minute period than PreK students in comparison classrooms. This represents a large 
effect size of about .72 standard deviations. Furthermore, as shown in Figure IV.5 below, by the end of the school year, the 
Minnesota Reading Corps PreK students’ average score of 8.0 alliterations reached the Spring benchmark of 8, whereas the 
comparison PreK students’ average score of 5.3 sounds was well below the benchmark. 
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Figure IV.5. Growth in Alliteration Fluency for 4- and 5-year old PreK program and comparison students 


9 
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Program 

Comparison 
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Benchmark 


Overall Findings for 3-year Old Students 

The Minnesota Reading Corps PreK program had a statistically significant and positive effect on 3-year old students' emergent 
literacy skills on two of the four IGDI measures developmental^ appropriate for 3-year old students. 46 As described in Chapter II, 
similar assessments were administered to the 3-year old PreK students as for 4- and 5-year old students, with the exception of 
letter sound fluency, which is not considered developmental^ appropriate for 3-year old children. Also, the letter name 
assessment for 3-year old students was untimed. 47 

Our analysis was conducted by age, due to developmental differences between 3-year old PreK students and 4- and 5-year old 
PreK students. However, it is important to note the substantially smaller sample size for the analysis of 3-year old students’ 
emergent literacy outcomes compared to that for 4- and 5-year old students. Smaller sample sizes substantially reduce the 
power to detect statistically significant differences, and the study was not designed to conduct these analyses with sufficient 
power. The lower sample size for 3-year old students is due to the fact that most Minnesota Reading Corps sites had classrooms 


46 Letter sound fluency was not administered to 3-year old students. 

47 As mentioned previously, the 4- and 5- year old version of the letter name assessment was a fluency measure. 
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that either only enrolled 4- and 5-year old students or a mix of 3-, 4-, and 5-year old students. Thus, these smaller sample sizes 
are one possible explanation for the lack of significance found on two of the four IGDI measures. 

Below we provide a discussion of the findings for 3-year old students, and Table IV.1 presents a summary of the data. As we did 
for the 4- and 5-year old findings above, several figures are provided, which show the average scores on the Fall, Winter, and 
Spring benchmarks for PreK students in Minnesota Reading Corps program and comparison classrooms. 


Table IV.2. Average Growth Among 3-Year Old PreK Students by Spring 


IGDI Assessment 

Treatment 

Assignment 

N 

Average 

Growth 

SE 

Effect Size 

Confidence Interval 


Program+ 

193 

3.17*** 

0.35 




Rhyming Fluency 

Comparison 

114 

0.86 

0.48 





Difference 

307 

2.30*** 

0.37 

0.43 

0.28 

0.59 


Program 

194 

6.52*** 

0.50 




Letter Name Recognition 

Comparison 

115 

5.11*** 

1.02 





Difference 

309 

1.41 

1.41 

- 

- 

- 


Program 

196 

6.94*** 

0.44 




Picture Name Fluency 

Comparison 

105 

3.43** 

0.88 





Difference 

301 

3.52** 

1.06 

0.42 

0.14 

0.71 


Program 

189 

1 .96*** 

0.22 




Alliteration Fluency 

Comparison 

189 

0.54 

0.66 





Difference 

281 

1.43 

0.68 

- 

- 

- 


’**<.001, **<.01, *<.05; +Minnesota Reading Corps 


What is the effect of the Minnesota Reading Corps program on 3-year old students’ rhyming fluency? 

The Minnesota Reading Corps program had a significant and large effect on 3-year old PreK students' rhyming fluency scores 

between Fall benchmark (September 2013) and Spring benchmark (May 2014). As shown in Table IV.2 above, by the end of the 

school year, PreK students in Minnesota Reading Corps classrooms could identify an average of 2.3 more rhyming words 

correctly in a two minute period than PreK students in comparison classrooms. This represents a substantial effect size of about 

.43 standard deviations. Although we provide an analysis of students’ progress relative to the benchmark target scores for the 4- 

and 5-year old PreK students, due to the limited research on appropriate target scores for 3-year old students, we focused on 

growth only at that age level. However, it is worth noting in Table IV.2 above that the 3-year old students in Minnesota Reading 

Corps program classrooms demonstrated significant growth between Fall and Spring Benchmark while students in the 

comparison classrooms did not. 
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Figure IV.6. Growth in Rhyming Fluency for 3-year old PreK program and comparison students 
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What is the effect of the Minnesota Reading Corps program on 3-year old students’ letter name 
recognition? 

Although large growth effects were found for 3-year old students in rhyming fluency, 3-year old PreK students in Minnesota 
Reading Corps classrooms did not attain significantly higher letter name recognition scores at Spring benchmark than did PreK 
students in comparison classrooms. By the end of the school year, the 3-year old Minnesota Reading Corps PreK students 
correctly identified an average of 1.4 more letters than PreK students in comparison classrooms. PreK students in both 
Minnesota Reading Corps program and comparison classrooms experienced significant growth between Fall and Spring 
benchmark in their average letter name recognition scores (6.5 and 5.1 letter names, respectively). Also, the average scores at 
Spring benchmark for 3-year old students in Minnesota Reading Corps program and comparison classrooms, shown in Figure 
IV.7, were similar (10.9 and 9.5, respectively). 
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Figure IV.7. Growth in Letter Name Recognition for 3-year old PreK program and comparison students 



■ Program 
Comparison 


What is the effect of the Minnesota Reading Corps program on 3-year old students’ picture naming 
fluency? 

Three year old PreK students in Minnesota Reading Corps classrooms attained significantly higher growth in picture naming 
fluency scores between Fall and Spring benchmark than did PreK students in comparison classrooms. The picture naming 
assessment measures child vocabulary and is an important indicator of future reading comprehension proficiency. As shown in 
Table IV.2, by the end of the school year, the 3-year old Minnesota Reading Corps program students correctly identified twice as 
many picture names as PreK students in comparison classrooms (6.9 and 3.4, respectively), resulting in a difference of 3.5 more 
pictures named correctly. Similar to rhyming fluency, this represents a substantial effect size of about .42 standard deviations. 
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Figure IV.8. Growth in Picture Name Fluency for 3-year old PreK program and comparison students 



What is the effect of the Minnesota Reading Corps program on 3-year old students’ alliteration fluency? 

For the final outcome measure of alliteration fluency , 3-year old PreK students in Minnesota Reading Corps classrooms did not 

experience significantly higher growth in scores between Fall and Spring benchmarks than did PreK students in comparison 

classrooms. By the end of the school year, the 3-year old Minnesota Reading Corps PreK program students correctly identified 

an average of 1 .4 more alliterative words than PreK students in comparison classrooms. Importantly, while the difference in 

growth between program and control sites was not significantly different, Table IV.2 above shows that the 3-year old students in 

Minnesota Reading Corps classrooms demonstrated significant growth from Fall to Spring Benchmark, whereas students in the 

comparison classrooms did not. Also, the average alliteration fluency score at Spring benchmark for 3-year old students in 

Minnesota Reading Corps classrooms, shown in Figure IV.7, is almost twice as high as the average for students in comparison 

classrooms (3.1 and 1.7, respectively). 


OUTCOME EVALUATION OF THE MINNESOTA READING CORPS PREK PROGRAM 


Page 39 


The Corporation for National and Community Service \ 2015 


Figure IV.9. Growth in Alliteration Fluency for 3-year old PreK program and comparison students 
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■ Program 
Comparison 


Findings by Major Demographic Groups 

Our findings include a set of subgroup analyses of student study participants. The evaluation team identified subgroups for 
further examination based on the major demographic groups generally of interest in education research and also of interest to 
CNCS and the Minnesota Reading Corps program. We provide results by gender, race/ethnicity, and Dual Language Learner 
(DLL) status. 48 Subgroups were defined based on characteristics provided at baseline and each level was determined by 
individual characteristics (e.g., male vs. female; White and non-White; DLL vs. non-DLL). 

Below the evaluation team provides two types of subgroup analysis. First, we added the identified moderators to the ANCOVA 
model and jointly tested the effects to determine if there are differential results across subgroups for 4- and 5-year old students, 
implying that the Minnesota Reading Corps PreK program may be more or less beneficial for selected subgroups of students. 49 


48 We did not examine students by Free and Reduced Price Lunch (FRPL) status, which is a measure of socio-economic status, 
because these data are not often collected on preschool children; however, the majority of our sample attended schools with 
high percentages of children with FRPL status, so it is reasonable to assume that the majority of the children in our study would 
likely meet the eligibility criteria for FRPL. 

49 Due to small sample sizes, subgroup analyses were not conducted for 3-year old students. 
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Results from the first set of subgroup statistical tests are provided in Table IV.3 below. The results show no statistically 
significant differences in the effect of the program for most subgroup variables across outcome measures, with the exception of 
DLL for picture naming. That is, the Minnesota Reading Corps PreK program is equally effective for both boys and girls, White 
and non-White students, and Dual Language Learners or English speaking students. One difference was found between non- 
DLL and DLL students for picture name fluency, where DLL students appear to have benefited from the program more so than 
non-DLL students on this outcome. 


Table IV.3. Moderator effects of subgroup variables on 4- and 5-year old PreK students by Spring 


IGDI Fluency Assessment 

Gender (Female) 

Race/Ethnicity 

(Non-White) 

Dual Language Learner (DLL) 


1.04 

0.01 

1.14 

Letter Sounds 





(N=511) 

(N=242) 

(N=1 27) 


-0.16 

1.07 

1.16 

Rhyming 





(N=578) 

(N=289) 

(N=147) 


-0.39 

2.52 

0.33 

Letter Names 





(N=587) 

(N=289) 

(N=148) 


1.11 

2.26 

3.65** 

Picture Names 





(N=580) 

(N=289) 

(N=148) 


-0.69 

-0.24 

-1.18 

Alliteration 





(N=522) 

(N=241) 

(N=1 28) 


’*<.01 


For the second subgroup analysis, we subdivided the 4- and 5-year old data by major subgroup (e.g., males, females, etc.), 
estimated the differences between the Minnesota Reading Corps program and comparison groups, and conducted significance 
testing to examine differences in patterns of findings by major subgroup. The results from this second subgroup analysis should 
be interpreted with caution, in particular when the results are not significant because sub-sectioning the data results in much 
smaller sample sizes than may be sufficient to detect differences between the program and comparison groups. As discussed 
earlier, smaller sample sizes substantially reduce the power to detect statistically significant differences, and the study was not 
designed to conduct these analyses with sufficient power. The results of both subgroup analyses are presented below. 
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Gender 

Both types of subgroup analyses found no differences in the effects of the program for male and female students. The moderator 
test presented in Table IV.3 above showed no significant differences in effects by gender for 4- and 5-year old students across 
all five literacy measures (letter sound fluency; rhyming fluency, letter name fluency, picture name fluency, and alliteration 
fluency). When the sample was segmented by gender, significant findings persisted for 4- and 5-year old students for both males 
and females. Furthermore, Table IV.4 shows that the average difference between the program and comparison groups at Spring 
benchmark were relatively similar for males and females. Although differences between 4- and 5-year old students in Minnesota 
Reading Corps classrooms compared to students in comparison classrooms were slightly higher for female than male students 
in letter sound fluency and picture name fluency, most of the five outcome measures show similar amounts of growth in 
emergent literacy skills across both gender subgroups. 

Table IV.4. Average growth among male and female 4- and 5-year old PreK students by Spring 


Male Students Female Students 


IGDI Fluency 
Assessment 

Treatment 

Assignment 

N 

Average Growth 

SE 

N 

Average 

Growth 

SE 


Program+ 


7.54*** 

0.33 


8.95*** 

0.44 

Letter Sounds 

Comparison 


4.23*** 

0.63 


4.60*** 

0.78 


Difference 

583 

3.31** 

0.89 

511 

4.35*** 

1.15 


Program 


6.83*** 

0.32 


7.77*** 

0.33 

Rhyming 

Comparison 


3.22*** 

0.50 


4.32*** 

0.35 


Difference 

641 

3.62*** 

0.71 

578 

3.45*** 

0.59 


Program 


13.14*** 

0.48 


14.94*** 

0.61 

Letter Names 

Comparison 


8.78*** 

0.85 


10.96*** 

0.92 


Difference 

648 

4.36** 

1.20 

587 

3.98* 

1.44 


Program 


8.58*** 

0.37 


8.79*** 

0.00 

Picture Names 

Comparison 


5.09*** 

0.50 


4.20*** 

0.48 


Difference 

642 

3.49*** 

0.71 

580 

4.59*** 

0.66 


Program 


4.97*** 

0.29 


5.78*** 

0.31 

Alliteration 

Comparison 


1.97*** 

0.50 


3.46*** 

0.53 


Difference 

588 

3.00*** 

0.71 

522 

2.32** 

0.76 


’**<.001, **<.01, *<.05; +Minnesota Reading Corps 
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Race/Ethnicity 

Due to small sample sizes for individual racial and ethnic subgroups, we conducted our analysis by examining differences 
between White and non-White students, which allowed us to collapse all minority groups into one subgroup category. In our 
initial test, where we examined whether race/ethnicity is a significant moderator of program effects, we found that race/ethnicity 
was not a significant factor in explaining variations in all five of the student outcomes for 4- and 5-year old students. 

Consistent with this finding, few differences were found in the patterns of effects when we subgrouped the sample into White and 
non-White and compared these findings to the subgroup analysis of moderator effects. Table IV.5 shows significant average 
growth on four of the five outcome measures (all except letter sound fluency) among 4- and 5-year old students in Minnesota 
Reading Corps PreK classrooms compared to similar students in comparison classrooms, despite White or non-White subgroup 
membership. Despite a significant finding for White students and no significant finding for non-White students, the differences in 
actual growth in letter sound fluency between students in Minnesota Reading Corps classrooms and those in comparison 
classrooms is about the same among White and non-White students (3.5 in both subgroups). Therefore, the lack of significance 
in letter sound fluency among non-White students is most likely due to the smaller sample size for this subgroup compared to 
White students (N=242 and N=807, respectively). 

Although the findings were statistically significant for both White and non-White students on four out of the five outcomes 
measured, it is also worth noting that larger average differences in growth were found between non-White students in Minnesota 
Reading Corps program classrooms and comparison classrooms compared to White students in Minnesota Reading Corps 
program and comparison classrooms across most measures. Therefore, the Minnesota Reading Corps program appears to have 
produced greater gains among non-White students relative to White students. Differences in outcomes were particularly evident 
in picture name fluency, a critically important measure of child vocabulary and a strong predictor of future reading 
comprehension proficiency. Non-White 4- and 5-year old students in Minnesota Reading Corps program classrooms achieved a 
larger amount of growth in letter name fluency (5.7 more letter names among non-White students compared to 3.2 more letters 
among White students) and picture name fluency (5.9 more picture names among non-White students compared to 3.6 more 
pictures among White students) compared to White students. 


Table IV.5. Average growth among White and non-White 4- and 5-year old PreK students by Spring 





White Students 



Non-White Students 

IGDI Fluency 
Assessment 

Treatment 

Assignment 

N 

Average Growth 

SE 

N 

Average 

Growth 

SE 


Program+ 


8.16*** 

0.32 



8.07*** 

1.38 

Letter Sounds 

Comparison 


4.65*** 

0.59 



4.55** 

1.73 


Difference 

807 

3.51*** 

0.87 

242 


3.52 

2.60 


Program 


7.66*** 

0.32 



6.27*** 

0.61 

Rhyming 

Comparison 


4.34*** 

0.44 



1.89* 

0.71 


Difference 

874 

3.32*** 

0.70 

289 


4.39*** 

0.99 


Program 


13.90*** 

0.53 


14.24*** 

1.12 
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Letter Names 

Comparison 


10.72*** 

0.94 


8.54*** 

2.53 


Difference 

892 

3.18* 

1.40 

289 

5.70** 

2.94 


Program 


9.00*** 

0.32 


7.85*** 

0.96 

Picture Names 

Comparison 


5.40*** 

0.50 


2.00 

0.94 


Difference 

877 

3.59*** 

0.75 

289 

5.85*** 

1.37 


Program 


5.64*** 

0.24 


4.59*** 

0.59 

Alliteration 

Comparison 


2.84*** 

0.36 


2.03** 

1.09 


Difference 

823 

2.80*** 

0.55 

241 

2.56*** 

1.48 


’**<.001, **<.01, *<.05; +Minnesota Reading Corps 


Dual Language Learners 

Some differences in the effects of the program were found in both subgroup analyses for DLL and non-DLL students. The 
moderator test showed no significant differences in effects by DLL status for 4- and 5-year old students in four of the five 
emergent literacy measures. Only one outcome measure, picture name fluency, was significant (p<01). 

When the student sample was divided into DLL and non-DLL, significant findings persisted for 4- and 5-year old students for both 
DLL and non-DLL on two of five measures. Table IV.6 shows that the average difference between the Minnesota Reading Corps 
program and comparison groups at Spring benchmark were significant for DLL and non-DLL students in rhyming and picture 
name fluency. The significance for picture name fluency may appear to contradict the finding from the first subgroup analysis, 
which indicates that DLL status is a significant moderator of program effects in picture naming. However, the substantial 
difference in size of the effect (difference) by subgroup (3.8 for non-DLL and 7.5 for DLL) are likely responsible for the significant 
finding in the moderator test. Growth in picture name fluency scores is particularly important for DLL students, as it is a measure 
of English vocabulary. 

Outcome measures in letter sound fluency , letter name fluency , and alliteration were significant for non-DLL students, but not for 
DLL students. However, on all three of these measures, students who attended Minnesota Reading Corps PreK sites were found 
to grow significantly more over the course of the year than their peers at comparison sites. Therefore, substantially smaller 
sample sizes for DLL students compared to non-DLL students may be a contributing factor to the lack of significance found on 
these measures. 


Table IV.6. Average growth among DLL and non-DLL 4- and 5-Year old PreK students by Spring 





Non-DLL Student 



DLL Students 


IGDI Fluency 

Treatment 





Average 


Assessment 

Assignment 

N 

Average Growth 

SE 

N 

Growth 

SE 


Program+ 


8.27*** 

0.30 


8.17*** 

1.38 

Letter Sounds 

Comparison 


4.54*** 

0.56 


3.3 

1.73 


Difference 

967 

3.73*** 

0.83 

127 

4.87 

2.60 


Program 


7.55*** 

0.26 


6.05*** 

0.61 
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Rhyming 

Comparison 


4.03*** 

0.36 


1.37 

0.71 


Difference 

1072 

3.52*** 

0.57 

147 

4.68*** 

0.99 


Program 


14.08*** 

0.49 


13.70*** 

1.12 

Letter Names 

Comparison 


9.94*** 

0.78 


9.22** 

2.53 


Difference 

1087 

4.14** 

1.20 

148 

4.47 

2.94 


Program 


8.94*** 

0.31 


8.22*** 

0.96 

Picture Names 

Comparison 


5.13*** 

0.43 


0.76 

0.94 


Difference 

1074 

3.81*** 

0.69 

148 

7.46*** 

1.37 


Program 


5.57*** 

0.18 


3.8*** 

0.59 

Alliteration 

Comparison 


2 ~i' | *** 

0.29 


2.11 

1.09 


Difference 

982 

2.86*** 

0.43 

128 

1.68 

1.48 


***<.001, **<.01, *<.05; +Minnesota Reading Corps 


Site Level Effects 

An additional analysis of interest to CNCS is the examination of whether student assessment scores varied as a function of the 
type of site the student attended (i.e., public school or Head Start center). 50 We were only able to examine differences for public 
schools and Head Start programs because only one pair of schools in the sample (one program and one comparison school) 
was community-based. For this analysis, we used a similar approach as for the second subgroup analysis for student-level 
demographics, whereby we subdivided the 4- and 5-year old data by school type (public school or Head Start center), estimated 
the differences between program and comparison groups, and conducted significance testing. Finally, we examined whether 
student outcomes varied by site by calculating an intraclass correlation (ICC) across pairs of matched sites (program and 
comparison). 

Site Type 

As shown in Table IV.7, significant effects of the program were found regardless of the type of site where the Minnesota Reading 
Corps program was being implemented. Students attending Minnesota Reading Corps classrooms in both public schools and 
Head Start centers showed significantly higher growth across all five outcome measures than their peers in comparison 
classrooms. While most differences on individual outcome measures were similar, 4- and 5-year old students in Head Start 
centers tended to show greater growth in letter name fluency than average public school students in Minnesota Reading Corps 
classrooms (6.2 and 3.5 letter names, respectively) relative to their peers in comparison classrooms. That said, the standard 
error for the difference in letter name fluency between program and comparison classrooms was comparatively large, suggesting 
significant variability in this outcome across sites. 

Also of interest, students attending Minnesota Reading Corps Head Start Centers demonstrated significantly greater growth in 
letter sound fluency and alliteration fluency than peers at comparison Head Start Centers. Students at Minnesota Reading Corps 


50 It is important to note that students enrolled in Head Start centers must be eligible for Free and Reduced Price Lunch (FRPL), a measure of 
poverty; whereas, public school students vary in their eligibility for FRPL. As such, children attending Head Start programs may be qualitatively 
different than those enrolled in public school programs. 
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Head Start program sites identified significantly more letter sounds by the end of the school year than at the beginning (7.07), 
whereas students at Head Start comparison sites did not identify significantly more letter sounds from the beginning of the school 
year (2.14). Likewise, students at Minnesota Reading Corps Head Start program sites identified significantly more alliterations 
by the end of the school year than at the beginning (4.78), whereas as students at Head Start comparison sites did not (1.13). 
These results show that the Minnesota Reading Corps program significantly improved growth in letter sound fluency and 
alliteration fluency at Head Start sites. 


Table IV.7. Average growth among 4 and 5-Year old PreK students at Public Schools and Head Start Centers by Spring 




Public School 


Head Start Cente 


IGDI Fluency 

Treatment 





Average 


Assessment 

Assignment 

N 

Average Growth 

SE 

N 

Growth 

SE 


Program+ 


8.44*** 

0.45 


7.07*** 

0.42 

Letter Sounds 

Comparison 


4.86*** 

0.68 


2.14 

1.32 


Difference 

841 

3.58** 

1.13 

211 

4.93** 

1.28 


Program 


7.40*** 

0.31 


7.14*** 

0.3 

Rhyming 

Comparison 


3.66*** 

0.39 


3.79*** 

0.56 


Difference 

919 

3.74*** 

0.67 

248 

3.35** 

0.83 


Program 


14.07*** 

0.58 


14.5*** 

0.85 

Letter Names 

Comparison 


10.61*** 

0.89 


8.32*** 

1.27 


Difference 

937 

3.47* 

1.46 

245 

6.18* 

2.11 


Program 


8.83*** 

0.38 


8.42*** 

0.37 

Picture Names 

Comparison 


4.78*** 

0.45 


3.52** 

0.61 


Difference 

921 

4.05*** 

0.82 

249 

4.91** 

0.93 


Program 


5.46*** 

0.22 


4.78*** 

0.26 

Alliteration 

Comparison 


3.03*** 

0.29 


1.13 

0.88 


Difference 

855 

2.43*** 

0.49 

211 

3.65* 

1.13 


***<.001, **<.01, *<.05; +Minnesota Reading Corps 


Site Pair-Level ICCs 

An additional analysis of interest to CNCS is the examination of whether student differences in growth varied as a function of the 
site pair (program or comparison) they attended. We examined this question by calculating site pair-level intraclass correlations 
(ICC) using the differences in students' Spring benchmark scores to determine whether the student outcome data indicated 
substantial variation in site pair-level effects. Intraclass correlations measure the portion of the total variation in an outcome that 
is associated with various levels of a variable, such as site assignment. In this case, the site pair-level ICC measures the degree 
to which students within the same site pair (program and comparison) correlate on their program effects (i.e., differences in 
growth). 

As shown in Table IV.8, the site pair-level ICCs are relatively low or not significant, indicating they are not meaningful. The ICCs 
for program effects vary by outcome measure, with the only statistically reliable ICCs found among 4- and 5-year old students in 
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letter sound fluency , picture name fluency , and alliteration fluency. However, the ICCs are relatively small at .06, .06, and .04, 
respectively. Therefore, it is possible to conclude that there is no significant variation in program effects across pairs of sites. 


Table IV.8. Site pair intraclass correlations for Spring-Fall differences 



3-year old students 
(N= 286) 

4- and 5-year old students 
(N= 1,480) 

IGDI Fluency Assessment 

ICC 

SE 

ICC 

SE 

Letter Sounds 

- 

- 

0.055* 

0.022 

Rhyming Fluency 

0.011 

0.025 

0.026 

0.013 

Letter Names 

0.001 

0.017 

0.083 

0.029 

Picture Names 

0.047 

0.036 

0.060* 

0.023 

Alliteration 

0.046 

0.036 

0.038* 

0.017 
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Conclusions 

The findings from the Minnesota Reading Corps PreK outcome evaluation provide important evidence for addressing the studies’ 
key research questions (presented in Chapter III). Below, the evaluation team offers our conclusions based on these findings and 
organizes them by major research questions. Following our assessment of the questions is a discussion on the implications of 
our findings for the Minnesota Reading Corps program. 

1. What is the effect of the Minnesota Reading Corps program on student literacy outcomes? 

The results of our analysis showed that preschool students who attended Minnesota Reading Corps classrooms achieved 
significantly higher emergent literacy assessment scores by the end of the school year than did students who did not attend a 
Minnesota Reading Corps classroom (see Chapter IV for more detail on these findings). The Minnesota Reading Corps PreK 
program had statistically significant and large effects across all five IGDI outcome measures of emergent literacy skills for 4- and 
5-year old students and on two of the four IGDI measures appropriate for assessing 3-year old students. 

The Minnesota Reading Corps PreK program had statistically significant effects across all five IGDI outcome measures, letter 
sound and letter name fluency (alphabet knowledge), rhyming fluency and alliteration fluency (phonological awareness), and 
picture name fluency (vocabulary), for 4- and 5-year old students between Fall benchmark (September 2013) and Spring 
benchmark (May 2014) . In addition, substantial effect sizes were found on each measure, ranging from .40 to .72, indicating that 
the Minnesota Reading Corps program has a comprehensive, positive effect on students’ emergent literacy skills. 

Despite the substantially smaller sample size for 3-year old students, the Minnesota Reading Corps program was found to have 
a significant and substantial effect on 3-year old PreK students’ rhyming fluency (phonological awareness) and picture name 
fluency (vocabulary) scores between Fall and Spring benchmark. Although growth was also found for 3-year old students on the 
letter name recognition (alphabet knowledge) and alliteration fluency (phonological awareness) outcome measures, 3-year old 
PreK students in Minnesota Reading Corps classrooms did not attain significantly higher letter name recognition and alliteration 
fluency scores at Spring benchmark than PreK students in comparison classrooms. The lack of statistical significance in letter 
name recognition is likely due to the fact that both the program and comparison PreK students experienced significant growth in 
their average letter name recognition scores. Letter name recognition the most widely emphasized emergent literacy skills in 
early childhood education. As such, it was not surprising to observe equally significant growth at both the Minnesota Reading 
Corps program and comparison sites. 

With respect to alliteration, 3-year old PreK students in neither Minnesota Reading Corps program classrooms nor comparison 
classrooms experienced significant growth in their scores between Fall and Spring benchmarks. Alliteration is typically a later 
developing phonological awareness skill, one which teachers may wait to engage with until children master precursor skills such 
as rhyming. The fact that large, statistically significant effects were found in alliteration fluency among 4- and 5- year old children 
at Minnesota Reading Corps sites strongly suggests that the program can effectively influence this skill but that it most 
developmentally appropriate for children older than 3-years of age. 
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Overall, the results of the student outcome analysis revealed that the Minnesota Reading Corps PreK Program is achieving its 
goal of creating Kindergarten-ready 4- and 5-year old students, as measured against the IGDI Spring benchmarks. Of particular 
interest is the large, significant effect that the Minnesota Reading Corps program produces in the growth in children’s oral 
language, specifically vocabulary as measured by the picture name fluency measure. Vocabulary is one of the strongest 
predictors of later reading comprehension. The Minnesota Reading Corps PreK program’s attention to not only decoding skills 
(i.e., letter name recognition and letter sound correspondence) but also oral language skills is a key program feature and likely a 
primary reason for the demonstrated effectiveness of the program. As witnessed during our Phase II PreK Process 
Assessment, 51 the program’s emphasis on intentionally integrating decoding and oral language skills in all aspects of daily 
interactions, as well as whole class, small group, and individualized instruction, results in numerous opportunities for children to 
connect text and language with personally meaningful content and experiences. This contextualized language and literacy 
instruction, as best evidenced by extended adult-child discourse and rich dialogic reading sessions, is a proven method for 
growing children’s emergent literacy skills. 

2. Does the effect on literacy outcomes vary by student characteristics/demographics? 

After establishing the overall positive effect of the Minnesota Reading Corps program on preschool students’ emergent literacy 
outcomes, we conducted analyses to examine whether differential effects of the program existed for specific subgroups of 4- and 
5-year old students based on the following student characteristics: gender (male/female), race/ethnicity (White/non-White), and 
Dual Language Learner (DLL) status (yes/no). For the most part, students from all subgroups examined produced significant 
growth in emergent literacy outcomes regardless of gender, race/ethnicity, or DLL status. In the instances where differential 
growth rates were observed, students who attended Minnesota Reading Corps PreK sites were often found to grow significantly 
more over the course of the year than their peers at comparison sites. Of note, students typically at risk for not being 
Kindergarten-ready, non-Whites and Dual Language Learners, posted significantly greater growth in vocabulary, as measured by 
picture name fluency, at Minnesota Reading Corps sites than at comparison sites. In fact, the vocabulary of non-White students 
and DLL students at comparisons sites did not grow significantly from Fall to Spring. Given the importance of vocabulary as a 
powerful predictor of later reading comprehension proficiency, the significant growth among non-White and DLL students at 
Minnesota Reading Corps PreK sites is a promising finding. It implies that the program elements that together comprise the 
Minnesota Reading Corps PreK program have a significant and meaningful impact on a critical skill (vocabulary) at program sites 
that is otherwise unaffected at comparison sites. Thus, two key conclusions for the evaluation are that the PreK program appears 
to have a positive effect on all students’ emergent literacy skills and that the program may be particularly effective in improving 
non-White and DLL students' vocabulary skills. 

3. Do effects on literacy outcomes vary by site/school? 

The Minnesota Reading Corps program appears to be consistently effective no matter the type of school setting in which the 
program occurs, either public school or Head Start center. Students enrolled at both types of PreK programs evidenced greater 


51 Hafford, C., Markovitz, C., Hernandez, M.W., Diaconis, A., Estrera, E., & Muyskens, P. (2014). Process Assessment of the Minnesota 
Reading Corps PreK Program: Phase II Extension. (Prepared under contract to the Corporation for National and Community Service). Chicago, 
IL: NORC at the University of Chicago. 
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growth in emergent literacy outcomes at Minnesota Reading Corps program sites than comparison sites. Of note, students at 
Minnesota Reading Corps Head Start centers achieved significantly greater growth on letter sound fluency and alliteration 
fluency than their peers at comparison site Head Start centers who did not improve on these measures during the school year. 
These results suggest that not only is the Minnesota Reading Corps PreK program effective in both public schools and Head 
Start centers, but that the program may be particularly effective in developing some emergent literacy skills for students enrolled 
in Head Start centers. 

The ICC findings indicate that the PreK program is effective across the individual sites where the students received Minnesota 
Reading Corps services. The low levels of variability in student outcome scores corroborate the high levels of fidelity of program 
implementation we observed in the Phase II Process Assessment. 52 The Minnesota Reading Corps program’s high quality 
training and program resources, use of validated formative assessments (i.e., IGDIs) to inform instruction, multi-layered 
supervisory structure, ongoing professional development and coaching sessions, and clear, objective method for measuring 
fidelity (i.e., ELLCO) are key program features that have led to such low levels of variability. These results support the conclusion 
that the Minnesota Reading Corps program is replicable in multiple school settings. 

Program Implications from Conclusions 

Overall, the results of this study show that the Minnesota Reading Corps PreK program is an effective model for improving 
preschool students' emergent literacy skills. Specifically, 4- and 5-year old students in Minnesota Reading Corps classrooms 
demonstrated significantly higher growth in their emergent literacy skills compared to their peers in comparison classroom across 
all five literacy measures. Furthermore, 3-year old students showed significantly higher growth on two of the four IGDI measures 
used to assess younger students. For the most part, the program was equally effective for all students and did not vary by 
gender, race, or DLL status. Likewise, for the most part, no major differences by site location or school type (public school or 
Head Start center) were found. In instances where differences were found, they favored at-risk students (i.e., non-White 
students, Dual Language Learners, Head Start centers). As such, the results show that the program is both effective and highly 
replicable in multiple settings with different types of students. 

The Minnesota Reading Corps PreK program model combines the human capital from at least 9 months of AmeriCorps service 
with evidence-based program elements, including a Rtl approach to differentiated instruction, regular formative assessment, the 
Literacy Rich Schedule, the SEEDS approach to high quality adult-child interactions, and intensive small-group and one-to-one 
intervention with Tier 2 and 3 students. 

The Minnesota Reading Corps PreK program’s comprehensive nature, including both whole class instructional strategies and 
differentiated interventions, results in class-wide impacts. The coordination between the two forms of instruction is likely a key 
source of the accelerated growth we observed in student emergent literacy outcomes. As outlined in the Phase II Process 
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Assessment, 53 the Minnesota Reading Corps program provides a common infrastructure for coordinated, intentional instruction 
through the Literacy Rich Schedule, a common pedagogy for interacting with PreK students via SEEDS, and proven strategies 
for embedding language and literacy instruction at times during the school day that are often overlooked (e.g., arriving at the 
beginning of the day, lining up for bathroom breaks or the bus, transitioning between circle time and choice time). Because the 
Minnesota Reading Corps PreK program is comprehensive, its success requires a strong commitment from site administrators 
and lead teachers. Furthermore, the program is most effective when all adults in a classroom coordinate their practice; therefore, 
it is critical that the lead teacher understand and fully commit to the Minnesota Reading Corps PreK model. 

The Minnesota Reading Corps program's use of a multi-layered supervisor structure, including site-based Internal Coaches and 
program-provided Master Coaches, effectively supports the implementation of such a comprehensive program. The PreK 
program’s focus on objective methods for monitoring both student and AmeriCorps member performance (i.e., IGDI assessments 
and ELLCO tool, respectively) supports fidelity of program implementation and further enforces a strong commitment to continual 
improvement and professional development throughout the year. In sum, the complete Minnesota Reading Corps PreK program, 
implemented with fidelity, is both highly replicable and effective in improving diverse PreK students' emergent literacy outcomes. 

Recommendations for Future Research 

Future research should more deeply examine differences in student outcomes across the model used at various PreK sites. 
Differences in the type and mix of staff used to support a PreK program are both interesting and relevant to assisting in bringing 
the Reading Corps program to scale. Whether the site includes Educator Corps only, Community Corps only, or a mix of the two 
has potential effects on student outcomes that could not be fully explored in this analysis. 

Future research should also explore the degree to which the impact of the Reading Corps model is seen in PreK sites that are 
relatively new to the Reading Corps (i.e., 1-3 years), versus sites that have been in the Reading Corps for a longer period time 
(i.e., 4+ years). The Reading Corps program provides additional intensity of service to recently established sites, and it might be 
helpful to the program if they could better understand at what point this additional intensity may no longer be needed, so 
resources can be more efficiently allocated to higher needs sites. 

One difficulty in completing this study was the degree of saturation of the PreK Reading Corps model and SEEDS training across 
PreK sites in Minnesota. Potential comparison sites were challenging to identify, especially in the only urban area in the state 
(i.e., Minneapolis-St. Paul). When potentially good candidate were identified, they were often found to have been “contaminated” 
because the site staff had previously received SEEDS training, a foundational component of the Reading Corps model. While the 
widespread dissemination of the effective SEEDS training is beneficial for Minnesota preschoolers, it was a significant hurdle in 
identifying comparison sites for the current study. Future research could replicate this study design in other states, where the 
presence of the Reading Corps PreK program and prevalence of SEEDS training is less widespread. This would allow for a 
potentially larger sample size, permitting additional points of inquiry to explore questions such as the impact of the program in 
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year one, two, and three separately (versus later years), differences in effects across staffing models, the program's effect 
across a broader range of student demographics, and the effect of the program model in states with less rigorous standards for 
early childhood education. 
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